eshangj/stackoverflow_q_and_a_sample huggingface dataset:
Question: below is the code i used to play multiple videos in parallel using multi threading pool. but only one video is playing for each input. i want each video to open separately. not combined import concurrent.futures RTSP_URL = "rtsp://wowzaec2demo.streamlock.net/vod/mp4:BigBuckBunny_115k.mp4" RTSP_List = [RTSP_URL, RTSP_URL, RTSP_URL, RTSP_URL] def url_to_video(url): video = cv2.VideoCapture(url) while True: _, frame = video.read() cv2.imshow("RTSP", frame) k = cv2.waitKey(1) if k == ord('q'): break video.release() cv2.destroyAllWindows() while True: with concurrent.futures.ThreadPoolExecutor() as executor: executor.map(url_to_video, RTSP_List)``` how to play each video separately. 

Answer: you just need each thread to use a different name for the window in cv2.imshow, so that each thread will generate a different window, and you should place them somewhere distinct so that they aren't appearing one over the other, i just added in index to them so that each distinct index will have a position on screen and different title, also you shouldn't destroy all windows when one is done ... import concurrent.futures import cv2 RTSP_URL = r"C:/Users/Ahmed/Desktop/test.mp4" RTSP_List = [(RTSP_URL,0), (RTSP_URL,1), (RTSP_URL,2), (RTSP_URL,3)] def url_to_video(tup): url,index = tup video = cv2.VideoCapture(url) while True: _, frame = video.read() cv2.imshow(f"RTSP {index}", frame) cv2.moveWindow(f"RTSP {index}", index*300, 0) k = cv2.waitKey(1) if k == ord('q'): break video.release() while True: with concurrent.futures.ThreadPoolExecutor() as executor: executor.map(url_to_video, RTSP_List) cv2.destroyAllWindows()  This code works on Windows because win32 backend allows each window to belong to a thread, for other operating systems and backends you can use the following non-threaded code. import cv2 import asyncio RTSP_URL = r"C:/Users/Ahmed/Desktop/test.mp4" RTSP_List = [(RTSP_URL,0), (RTSP_URL,1), (RTSP_URL,2), (RTSP_URL,3)] break_all = False async def url_to_video(tup): global break_all url,index = tup video = cv2.VideoCapture(url) while True: result, frame = await asyncio.get_event_loop().run_in_executor(None, video.read) if not result: break cv2.imshow(f"RTSP {index}", frame) cv2.moveWindow(f"RTSP {index}", index*300, 0) k = cv2.waitKey(1) if k == ord('q'): break_all = True if break_all: break video.release() async def main(): global break_all break_all = False await asyncio.gather(*[url_to_video(x) for x in RTSP_List]) while True: asyncio.run(main()) cv2.destroyAllWindows()
Question: So I have a Polars dataframe looking as such df = pl.DataFrame( { "ItemId": [15148, 15148, 24957], "SuffixFactor": [19200, 200, 24], "ItemRand": [254, -1, -44], "Stat0": ['+5 Defense', '+$i Might', '+9 Vitality'], "Amount": ['', '7', ''] } )  I want to replace $i in the column "Stat0" with Amount whenever Stat0 contains i$ I have tried a couple different things such as: df = df.with_columns( pl.col('Stat0').str.replace(r'\$i', pl.col('Amount')) )  Expected result result = pl.DataFrame( { "ItemId": [15148, 15148, 24957], "SuffixFactor": [19200, 200, 24], "ItemRand": [254, -1, -44], "Stat0": ['+5 Defense', '+7 Might', '+9 Vitality'], "Amount": ['', '7', ''] } )  shape: (3, 5) ┌────────┬──────────────┬──────────┬─────────────┬────────┐ │ ItemId ┆ SuffixFactor ┆ ItemRand ┆ Stat0 ┆ Amount │ │ --- ┆ --- ┆ --- ┆ --- ┆ --- │ │ i64 ┆ i64 ┆ i64 ┆ str ┆ str │ ╞════════╪══════════════╪══════════╪═════════════╪════════╡ │ 15148 ┆ 19200 ┆ 254 ┆ +5 Defense ┆ │ │ 15148 ┆ 200 ┆ -1 ┆ +7 Might ┆ 7 │ │ 24957 ┆ 24 ┆ -44 ┆ +9 Vitality ┆ │ └────────┴──────────────┴──────────┴─────────────┴────────┘  But this doesn't seem to work. I hope someone can help. Best regards

Answer: As of Polars 0.14.4, the replace and replace_all expressions allow an Expression for the value parameter. Thus, we can solve this more simply as: df.with_columns( pl.col('Stat0').str.replace(r'\$i', pl.col('Amount')) )  shape: (3, 5) ┌────────┬──────────────┬──────────┬─────────────┬────────┐ │ ItemId ┆ SuffixFactor ┆ ItemRand ┆ Stat0 ┆ Amount │ │ --- ┆ --- ┆ --- ┆ --- ┆ --- │ │ i64 ┆ i64 ┆ i64 ┆ str ┆ str │ ╞════════╪══════════════╪══════════╪═════════════╪════════╡ │ 15148 ┆ 19200 ┆ 254 ┆ +5 Defense ┆ │ │ 15148 ┆ 200 ┆ -1 ┆ +7 Might ┆ 7 │ │ 24957 ┆ 24 ┆ -44 ┆ +9 Vitality ┆ │ └────────┴──────────────┴──────────┴─────────────┴────────┘
Question: I wish to select only columns with fewer than 3 unique values. I can generate a boolean mask via pl.all().n_unique() < 3, but I don't know if I can use that mask via the polars API for this. Currently, I am solving it via python. Is there a more idiomatic way? import polars as pl, pandas as pd df = pl.DataFrame({"col1":[1,1,2], "col2":[1,2,3], "col3":[3,3,3]}) # target is: # df_few_unique = pl.DataFrame({"col1":[1,1,2], "col3":[3,3,3]}) # my attempt: mask = df.select(pl.all().n_unique() < 3).to_numpy()[0] cols = [col for col, m in zip(df.columns, mask) if m] df_few_unique = df.select(cols) df_few_unique  Equivalent in pandas: df_pandas = df.to_pandas() mask = (df_pandas.nunique() < 3) df_pandas.loc[:, mask] 

Answer: The selected answer, though syntactically clean, is inefficient. You can do about better Let us first include at least two filters rather than just one Problem: Select only those columns where the number of unique values is between 1 and 200 The thing to consider is that you would need a pass over the data no matter what. So, reading it in is the first step Then, if you do pl.select( [s for s in df if s.n_unique() < 200 and s.n_unique() > 1] )  You are computing the filters in sequence and also keeping them in memory. Htop confirms that using just one core of the machine The ideal solution is to do it all in parallel. Let us do a few benchmarks. I am using a 32 cores machine. Parallelism would reduce the time further on machines with more cores set up the dataframes: import polars as pl import numpy as np df = pl.DataFrame({f'a_{i}':np.random.choice(['a','b','c','d'], 10000000) for i in range(100)})  This would take up about 20 GiB RAM. So, be careful if you want to replicate Selected solution (htop confirms that this solution uses only one core) %%timeit _df = pl.select( [s for s in df if s.n_unique() < 200 and s.n_unique() > 1] ) output: 18.7 s ± 92.6 ms per loop (mean ± std. dev. of 7 runs, 1 loop each)  Let us now try to run the filters in parallel (htop confirms) %%timeit _df = df.select((pl.all().n_unique() < 200) & (pl.all().n_unique() > 1)) output: 1.35 s ± 21.4 ms per loop (mean ± std. dev. of 7 runs, 1 loop each)  We are still computing every filter twice in the two .n_unique() calls above. Let us do with just one by using in_between (parallel execution - htop confirms) %%timeit _df = df.select((pl.all().n_unique().is_between(1,200))) output: 708 ms ± 21.3 ms per loop (mean ± std. dev. of 7 runs, 1 loop each)  Btw, if you don't want to remember the APIs like in_between and also not compute the n_unique() twice, you can use the lazy semantics df_lazy = df.lazy()  Now, try the above solution %%timeit _df = df_lazy.select((pl.all().n_unique() < 200) & (pl.all().n_unique() > 1)).collect() output: 718 ms ± 15.6 ms per loop (mean ± std. dev. of 7 runs, 1 loop each)
Question: When I call a coroutine without awaiting, in addition to a message warning me I have not awaited the coroutine, I also get the following warning message: RuntimeWarning: Enable tracemalloc to get the object allocation traceback  I know how to fix this i.e. by awaiting the coroutine (and I do see a lot of questions about this warning, but all the answers are how to fix it; my goal is to understand it; please don't mark my question as duplicate if possible :) ); in particular, what I'd really like to understand is:  What is tracemalloc? How do I enable it? What is the object allocation traceback? How is it related to coroutines? and, in particular Why is there a warning suggesting to enable tracemalloc when I don't await a coroutine?  My goal in asking this is understanding the details and inner-workings of asyncio better.

Answer:  What is tracemalloc? How do I enable it?  tracemalloc is a module that is used to debug memory allocation in Python. You can enable it by setting PYTHONTRACEMALLOC environment variable to 1. Check the Tracemalloc docs for more info.  What is the object allocation traceback?  It's a way how Python manages memory and allocates memory for its objects.  Why is there a warning suggesting to enable tracemalloc when I don't await a coroutine?  I guess this is because memory is allocated and never used properly so we get loss of resources, i.e. a memory leak.
Question: I have a dataframe:  lft rel rgt num 0 t3 r3 z2 3 1 t1 r3 x1 9 2 x2 r3 t2 8 3 x4 r1 t2 4 4 t1 r1 z3 1 5 x1 r1 t2 2 6 x2 r2 t4 4 7 z3 r2 t4 5 8 t4 r3 x3 4 9 z1 r2 t3 4  And a reference dictionary: replacement_dict = { 'X1' : ['x1', 'x2', 'x3', 'x4'], 'Y1' : ['y1', 'y2'], 'Z1' : ['z1', 'z2', 'z3'] }  My goal is to replace all occurrences of replacement_dict['X1'] with 'X1', and then compute the group-wise sum of num rows. For example, any instance of 'x1', 'x2', 'x3' or 'x4' will be replaced by 'X1', etc., and the total of the 'X1'-'r1'-'t2' group (which is created by the remapping above) is 6, etc. So my desired output is:  lft rel rgt num 0 X1 r3 t2 8 1 X1 r1 t2 6 2 X1 r2 t4 4 3 t1 r3 X1 9 4 t4 r3 X1 4  I am working with a dataframe with 6 million rows and a replacement dictionary with 60,000 keys. This is taking forever using a simple row wise extraction and replacement. How can this (specifically the last part) be scaled efficiently? Is there a pandas trick that someone can recommend?

Answer: Reverse the replacement_dict mapping and map() this new mapping to each of lft and rgt columns to substitute certain values (e.g. x1->X1, y2->Y1 etc.). As some values in lft and rgt columns don't exist in the mapping (e.g. t1, t2 etc.), call fillna() to fill in these values.1 You may also stack() the columns whose values need to be replaced (lft and rgt), call map+fillna and unstack() back but because there are only 2 columns, it may not be worth the trouble for this particular case. The second part of the question may be answered by summing num values after grouping by lft, rel and rgt columns; so groupby().sum() should do the trick. # reverse replacement map reverse_map = {v : k for k, li in replacement_dict.items() for v in li} # substitute values in lft column using reverse_map df['lft'] = df['lft'].map(reverse_map).fillna(df['lft']) # substitute values in rgt column using reverse_map df['rgt'] = df['rgt'].map(reverse_map).fillna(df['rgt']) # sum values in num column by groups result = df.groupby(['lft', 'rel', 'rgt'], as_index=False)['num'].sum()  1: map() + fillna() may perform better for your use case than replace() because under the hood, map() implements a Cython optimized take_nd() method that performs particularly well if there are a lot of values to replace, while replace() implements replace_list() method which uses a Python loop. So if replacement_dict is particularly large (which it is in your case), the difference in performance will be huge, but if replacement_dict is small, replace() may outperform map(). See this answer, which includes different benchmarks showing interaction between dictionary size and dataframe length, to get an idea of when to use replace and when to use map+fillna.
Question: I am working on an app that uses a rich.Progress for rendering progress bars. The problem is rich.prompt.Confirm just flashes instead of showing the message and asking for the confirmation while in the Progress context. Demo Code from rich.progress import Progress from rich.prompt import Confirm from time import sleep with Progress() as progress: task = progress.add_task('Cooking') while not progress.finished: if Confirm.ask('Should I continue', default=False): progress.update(task, advance=0.6) sleep(0.4)  EDIT: I have seen git issues and researched a bit and it seems input(which the rich.Prompt uses) doesn't work on any thing that uses rich.Live(which the rich.Progress uses). So now my question is, How can you structure your code so that you don't put a prompt inside a rich.Progress context manager. Or any possible workarounds to this issue.

Answer: So from the Github Issue (That might be the one you talked about), that is now a workaround, thanks to Leonardo Cencetti. The solution is simple. He pause the progress and clear the progress lines. When you are done, he starts the progress again. For Future people here is his code: from rich.progress import Progress class PauseProgress: def __init__(self, progress: Progress) -> None: self._progress = progress def _clear_line(self) -> None: UP = "\x1b[1A" CLEAR = "\x1b[2K" for _ in self._progress.tasks: print(UP + CLEAR + UP) def __enter__(self): self._progress.stop() self._clear_line() return self._progress def __exit__(self, exc_type, exc_value, exc_traceback): self._progress.start()  And in you MWC, it is going to be used like: from rich.progress import Progress from rich.prompt import Confirm from time import sleep with Progress() as progress: task = progress.add_task('Cooking') while not progress.finished: with PauseProgress(progress): ok_to_go = Confirm.ask('Should I continue', default=False) if not ok_to_go: break progress.update(task, advance=0.6) sleep(1)
Question: In a 2D tensor like so tensor([[0.8771, 0.0976, 0.8186], [0.7044, 0.4783, 0.0350], [0.4239, 0.8341, 0.3693], [0.5568, 0.9175, 0.0763], [0.0876, 0.1651, 0.2776]])  How do you sort the rows based off the values in a column? For instance if we were to sort based off the last column, I would expect the rows to be such... tensor([[0.7044, 0.4783, 0.0350], [0.5568, 0.9175, 0.0763], [0.0876, 0.1651, 0.2776], [0.4239, 0.8341, 0.3693], [0.8771, 0.0976, 0.8186]])  Values in the last column are now in ascending order.

Answer: t = torch.rand(5, 3) COL_INDEX_TO_SORT = 2 # sort() returns a tuple where first element is the sorted tensor # and the second is the indices of the sorted tensor. # The [1] at the end is used to select the second element - the sorted indices. sorted_indices = t[:, COL_INDEX_TO_SORT].sort()[1] t = t[sorted_indices]
Question: Is it possible to conditionally skip parametrized tests?Here's an example: @pytest.mark.parametrize("a_date", a_list_of_dates) @pytest.mark.skipif(a_date > date.today()) def test_something_using_a_date(self, a_date): assert <some assertion>  Of course I can do this inside the test method, but I'm looking for a structured way to do this with pytest.

Answer: If you create your own method you check the values in test collection time and run the relevant tests only a_list_of_dates = [date.today(), date(2024, 1, 1), date(2022, 1, 1)] def get_dates(): for d in a_list_of_dates: if d <= date.today(): yield d class TestSomething: @pytest.mark.parametrize("a_date", get_dates()) def test_something_using_a_date(self, a_date): print(a_date)  Output TestSomething::test_something_using_a_date[a_date0] PASSED [ 50%] 2022-08-24 TestSomething::test_something_using_a_date[a_date1] PASSED [100%] 2022-01-01  If you still want to see the skipped tests you can add the skip marker to the relevant tests def get_dates(): for d in a_list_of_dates: markers = [] if d > date.today(): markers.append(pytest.mark.skip(reason=f'{d} is after today')) yield pytest.param(d, marks=markers)  Output TestSomething::test_something_using_a_date[a_date0] PASSED [ 33%] 2022-08-24 TestSomething::test_something_using_a_date[a_date1] SKIPPED (2024-01-01 is after today) [ 66%] Skipped: 2024-01-01 is after today TestSomething::test_something_using_a_date[a_date2] PASSED [100%] 2022-01-01
Question: I was working with f-strings, and I am fairly new to python. My question is does the f-string formatting cast a variable(an integer) into a string? number = 10 print(f"{number} is not a string")  Is number cast into a string?

Answer: f"..." expressions format values to strings, integrate the result into a larger string and return that result. That's not quite the same as 'casting'*. number is an expression here, one that happens to produce an integer object. The integer is then formatted to a string, by calling the __format__ method on that object, with the first argument, a string containing a format specifier, left empty: >>> number = 10 >>> number.__format__('') '10'  We'll get to the format specifier later. The original integer object, 10, didn't change here, it remains an integer, and .__format__() just returned a new object, a string: >>> f"{number} is not a string" '10 is not a string' >>> number 10 >>> type(number) int  There are more options available to influence the output by adding a format specifier to the {...} placeholder, after a :. You can see what exact options you have by reading the Format Specification Mini Language documentation; this documents what the Python standard types might expect as format specifiers. For example, you could specify that the number should be left-aligned in a string that's at least 5 characters wide, with the specifier <5: >>> f"Align the number between brackets: [{number:<5}]" 'Align the number between brackets: [10 ]'  The {number:<5} part tells Python to take the output of number.__format__("<5") and use that as the string value to combine with the rest of the string, in between the [ and ] parts. This still doesn't change what the number variable references, however.  *: In technical terms, casting means changing the type of a variable. This is not something you do in Python because Python variables don't have a type. Python variables are references to objects, and in Python it is those objects that have a type. You can change what a variable references, e.g. number = str(number) would replace the integer with a string object, but that's not casting either.
Question: I'm struggling to install tensorflow with a M1 mac. I've got python 3.9.7 and Monterrey 12.3 and apple silicon visual studio code. There is an apple solution involving miniconda apple dependancies and tensorflow-macos and tensorflow-metal. However this solution is not good for me as I have to use Rosetta2 emulator for multiple packages including PyQt5 etc. I was wondering if anyone has been able to use their M1 macs and pip installed tensorflow in a venv rosetta terminal. Thank you. Kevin

Answer: Running TensorFlow on miniforge + conda-forge (arm64) TensorFlow can run natively on M1 (arm64) macs. A highly recommended, easy way to install TensorFlow on arm64 macs is to via conda-forge. You should install python via miniforge or miniconda, because there is an arm64 (Apple Sillicon) distribution. With this, as of today, you can install the latest version 2.10.0 of TensorFlow: $ lipo -archs $(which python3) # python3 is running natively as arm64 arm64 $ conda install -c conda-forge tensorflow  Note: tensorflow-macos 2.4.0 is obsolete so you shouldn't be using that. But still want Rosetta 2? Try conda-forge. If you really need to have python running on Rosetta 2 (x86_64) in cases where some packages does not support arm64, you can still install TensorFlow with a macOS x86_64 release via conda. Installing via pip and PyPI repository won't work here, because you will run into Illegal hardware instruction segfault because Google's official TF macos-x86_64 wheel releases on PyPI assumes a target platform that has AVX instructions. $ lipo -archs $(which python3) # x86_64 means Rosetta 2 x86_64 $ conda install -c conda-forge tensorflow # install via conda $ python -c 'import tensorflow; print(tensorflow.__version__)'
Question: I have a Pydantic class model that represents a foreign API that looks like this: class Position(BaseModel): AccountID: str AveragePrice: str AssetType: str Last: str Bid: str Ask: str ConversionRate: str DayTradeRequirement: str InitialRequirement: str PositionID: str LongShort: str Quantity: int Symbol: str Timestamp: str TodaysProfitLoss: str TotalCost: str MarketValue: str MarkToMarketPrice: str UnrealizedProfitLoss: str UnrealizedProfitLossPercent: str UnrealizedProfitLossQty: str  This is the names of the API endpoint that I need to point to. I simply want to change the pascal case fields to a pythonic design. What I want is to deserialize the foreign API and serialize it back using Pydantic's BaseModel class. My problem is that if I use Pydantic's class Fields like this: class Position(BaseModel): account_id: str = Field(alias='AccountID') average_price: str = Field(alias='AveragePrice') asset_type: str = Field(alias='AssetType') last: str = Field(alias='Last') bid: str = Field(alias='Bid') ask: str = Field(alias='Ask') conversion_rate: str = Field(alias='ConversionRate') day_trade_requirement: str = Field(alias='DayTradeRequirement') initial_requirement: str = Field(alias='InitialRequirement') position_id: str = Field(alias='PositionID') long_short: str = Field(alias='LongShort') quantity: int = Field(alias='Quantity') symbol: str = Field(alias='Symbol') timestamp: str = Field(alias='Timestamp') todays_profit_loss: str = Field(alias='TodaysProfitLoss') total_cost: str = Field(alias='TotalCost') market_value: str = Field(alias='MarketValue') mark_to_market_price: str = Field(alias='MarkToMarketPrice') unrealized_profit_loss: str = Field(alias='UnrealizedProfitLoss') unrealized_profit_loss_percent: str = Field(alias='UnrealizedProfitLossPercent') unrealized_profit_loss_qty: str = Field(alias='UnrealizedProfitLossQty')  I can only deserialize it and not the other way around. Any way I can do it for both "directions"?

Answer: Yes, it's possible, use .dict(by_alias=True), see example: from pydantic import BaseModel, Field class Position(BaseModel): account_id: str = Field(alias='AccountID') pos2 = Position(AccountID='10') print(pos2.dict()) print(pos2.dict(by_alias=True))  Output: {'account_id': '10'} {'AccountID': '10'}