eshangj/stackoverflow_q_and_a_sample huggingface dataset:
Question: I have 3 functions: func_1, func_2, and func_3. I would like to run these asynchronously, so that I do not have to wait for func_1 to finish before func_2 starts executing. The problem is, that the definition of func_1 for example looks something like this: async def func_1(a, b): x = some_sync_func(a) y = some_other_sync_func(b) z = another_sync_func(x, y) return yet_another_sync_func(z)  The functions that I am calling within func_1 are all synchronous functions which are non-awaitable. Thus, they will block the execution of func_2 and func_3. I read here that loop.run_in_executor() can be used to call synchronous functions from asynchronous functions without blocking the execution. Thus, I modified the definition of func_1 as follows: async def func_1(a, b): loop = asyncio.get_event_loop() x = await loop.run_in_executor(None, some_sync_func, a) y = await loop.run_in_executor(None, some_other_sync_func, b) z = await loop.run_in_executor(None, lambda: another_sync_func(a,b)) w = await loop.run_in_executor(None, yet_another_sync_func, z) return w  Is this the right way to deal with this problem? Am I using loop.run_in_executor() correctly? Here, the docs provide an example which seems to support this. I don't know what threads are, or what a "process pool" is, and haven't really been able to make much sense of the docs.

Answer: Almost right, but since you are awaiting eagerly at each function call, the next line of code in each case (after the await) will only be called when the line with await finishes execution. However if you call func_1 in parallel from some other place, two instances of func_1 will work in parallel. (I am almost sure that is not what you want). So, in order for these other functions to actually run in parallel (in other threads), you have to create the task to run each of them, but not await it immediately, instead, you gather all the tasks you want to run in parallel and await for them at once (usually with a function properly named gather ): ... async def func_1(a, b): loop = asyncio.get_event_loop() task_x = loop.run_in_executor(None, some_sync_func, a) task_y = loop.run_in_executor(None, some_other_sync_func, b) task_z = loop.run_in_executor(None, lambda: another_sync_func(a,b)) x, y, z = await asyncio.gather(task_x, task_y, task_z) # this depends on `z` so, it is not included in the gather. # if its return value is not important, you can ommit the # await, return the task, and await for it sometime later. w = await loop.run_in_executor(None, yet_another_sync_func, z) return w ...
Question: I have a base class: from abc import abstractmethod class Thing: @abstractmethod def _process(self): ... def process(self, x: int): self.pre_process(x) return self._process()  How do I typehint process as returning the return type of _process? My first thought was something like: from abc import abstractmethod from typing import TypeVar class Thing: T = TypeVar("T") @abstractmethod def _process(self) -> T: ... def process(self, x: int) -> T: ...  But mypy 1.3.0 complains quite rightly that T is only present once in the function signature:  > mypy /tmp/t.py ... error: A function returning TypeVar should receive at least one argument containing the same TypeVar ... 

Answer: You can make Thing inherit Generic[T]. from typing import TypeVar from typing import Generic from abc import abstractmethod T = TypeVar("T") class Thing(Generic[T]): @abstractmethod def _process(self) -> T: ... def process(self, x: int) -> T: return self._process()  > mypy /tmp/t.py Success: no issues found in 1 source file  Now you can inherit from Thing like this: class A(Thing[list]): def _process(self) -> list: return []
Question: For the following multifacet plot  df = pd.DataFrame({ 'row': [0,0,0,0,0,0,0,0,1,1,1,1,1,1,1,1], 'col': [0,0,0,0,1,1,1,1,0,0,0,0,1,1,1,1 ], 'x_value': [1,2,3,4,1,2,3,4,1,2,3,4,1,2,3,4], 'count': [1,7,4,0,0,3,1,3,1,9,2,2,0,0,3,4] }) df = df.query('count != 0 ') fig = px.bar(df, x='x_value', y='count', facet_col='col', facet_row='row', template='simple_white') fig.for_each_xaxis(lambda xaxis: xaxis.update(showticklabels=True, title_font = dict(size =20), type = 'category')) fig.show()  for each subplot, i want to show ONLY the x-ticks present in the data. The circled ticks should not be showing: 

Answer: To get what you are looking for, you need to customize each x-axis (and y-axis, if required) to include matches=None. This will stop trying to match the columns across each of the subplots. So, replace the line... fig.for_each_xaxis(lambda xaxis: xaxis.update(showticklabels=True, title_font = dict(size =20), type = 'category'))  with fig.for_each_xaxis(lambda xaxis: xaxis.update(showticklabels=True, matches=None, title_font = dict(size =20), type = 'category'))  and you will get below plot. Hope this is what you are looking for...
Question: I create a normal python list a = [[[1,2],[3,4]],[[5,6],[7,8]],[[9,10],[11,12]]]  Now I want to shift the first row of each 2x2 array to the previous 2x2 array, wrapping the first back to the last. I use the following unpacking assignment statement: a[0][0],a[1][0],a[2][0] = a[1][0],a[2][0],a[0][0]  I get the following, which is what I want print(a) [[[5, 6], [3, 4]], [[9, 10], [7, 8]], [[1, 2], [11, 12]]]  Now, I do the same thing, this time, using numpy arrays b = np.arange(1,13).reshape((3,2,2)) print(b) [[[ 1 2] [ 3 4]] [[ 5 6] [ 7 8]] [[ 9 10] [11 12]]]  The shift is the same as before but using numpy indexing syntax b[0,0],b[1,0],b[2,0] = b[1,0],b[2,0],b[0,0] print(b) [[[ 5 6] [ 3 4]] [[ 9 10] [ 7 8]] [[ 5 6] [11 12]]]  As you can see, this assignment left the first and last 2x2 arrays with the same first row. Would you expect this behavior difference? Why does numpy do this but not the regular list type? How could the numpy assignment be done to result in the same result as the list?

Answer: The way I reason this out is that since b is a (3,2,2) array each of b[1,0],b[2,0],b[0,0]  is a view of b. That is, 3 (2,) arrays, but each uses a different part of the b data_buffer. During the assignment, b[0,0] is set to the values at b[1,0]. By the time it assigns to b[2,0], B[0,0] now has the new values, [5,6]. I've seen this on occasion before. It's another case where arrays aren't exactly the same as lists. Lists contain pointers to lists, where as array indexing either produces copies or views. Views are convenient, but can mess with list/reference based intuitions. Fortunately arrays can assign several values at once, so we can do: b[[0,1,2],0] = b[[1,2,0],0]  or the equivalent with slices b[:,0]=b[::-1,0] Earlier questions focus more on the correct way of doing the swap, and less on why there's a difference. So won't mark this as duplicate Swap slices of Numpy arrays Row exchange in Numpy
Question: The purpose of the algorithm is to auto align-sheet music pages based on staves/systems content. The algorithm need to detect the bounding box  to allow to easily compute the left/right and top/bottom margin for the wholes pages. Current algorithm like morphological operations in openCV (using cv2.HoughLinesP for example) fails. I need to found the most precise coordinates at least the top (first) or bottom (last) staff line and at least the left (systems) lines to compute the bounding box (red lines at left of the picture) What is the state-of-the-art algorithm to work on this kind of document. The full resolution is in 600 dpi (5078 x 6566 pixels) if that can help (without downsizing). Thank you very much.

Answer: You already looked into "staff-line removers". For this task, you need a part of that: the part that identifies staff lines. Assuming the scan is upright, not rotated by a few degrees, that is usually done with morphology operations that use a "line" shaped kernel. To also drop the short thick vertical bars on the left, your suggestion of a column histogram allows you to overcome the short horizontal stubs that connected those bars to the long vertical line.  threshold  level, mask = cv.threshold( src=im, thresh=128, maxval=255, type=cv.THRESH_BINARY_INV | cv.THRESH_OTSU)    morphology  hkernel = np.ones((1, 3), dtype=np.uint8) vkernel = np.ones((3, 1), dtype=np.uint8) horizontals = cv.morphologyEx(src=mask, op=cv.MORPH_OPEN, kernel=hkernel, iterations=50) verticals = cv.morphologyEx(src=mask, op=cv.MORPH_OPEN, kernel=vkernel, iterations=40) combined = horizontals | verticals  Tweak the kernel sizes and number of iterations.   bounding box  bbox = cv.boundingRect(combined) canvas = cv.cvtColor(im >> 1, cv.COLOR_GRAY2BGR) cv.rectangle(img=canvas, rec=bbox, color=(0,255,0), thickness=1)  Includes the short stubs on the left edge.   column histogram to start the bounding box at the first long vertical  colhist = combined.mean(axis=0) / 255 colmax = colhist.max()   # option: find global maximum left = colhist.argmax() # option: find first above threshold indices, = np.where(colhist >= 0.5 * colmax) left = indices[0]  fix the box up: x0,y0,w,h = bbox x1 = x0 + w # right edge stays where it is bbox = (left, y0, x1 - left, h)    shift the image so that the box is centered  x0,y0,w,h = bbox bcenter = np.array([x0 + (w-1)/2, y0 + (h-1)/2]) icenter = np.array([(iw-1)/2, (ih-1)/2]) # image size T = np.eye(3) T[0:2, 2] = (+icenter - bcenter).round() canvas = cv.warpAffine( src=canvas, M=T[:2], dsize=(iw, ih), flags=cv.INTER_NEAREST, borderMode=cv.BORDER_REPLICATE)   This will be confused if there are any other long lines that aren't staff lines. That scan shows the bottom edge of the sheet of paper. You should make sure that either doesn't show up in the scan, or you need to erase image content near the edges (which is what I did here, without detailing it).
Question: I'm writing a Playwright test in Python. I have a table that I want to grab every row from to perform some actions within a for loop but I want to skip the first row. I am able to grab every row within a table by doing something as simple as the following: table = page.get_by_role("row").all() for row in table: print(row)  But I want it to skip the first row. So I know the css selector for that is :nth-child(n+1) but I'm not sure how to use this in Playwright. I tried doing something like: table = page.get_by_role("row:nth-child(n+1)")  But I got the error: unexpected symbol ":" Then I tried: table = page.get_by_role("row").nth("n+1")  But that also didn't work. Finally, tried this: table = page.locator("[role=row]:nth-child(n+1)")  But I got the error: ERROR test_table.py - TypeError: 'Locator' object is not iterable How do I properly query all the rows skipping the first one?

Answer: Rather than manipulating the selector, you simply start the for loop at the second element, like this: table = page.get_by_role("row").all() # Using slice notation for row in table[1:]: print(row)
Question: from fastapi import Depends, FastAPI class MyDependency: def __init__(self): # Perform initialization logic here pass def some_method(self): # Perform some operation pass def get_dependency(): # Create and return an instance of the dependency return MyDependency() app = FastAPI() @app.get("/example") def example(dependency: MyDependency = Depends(get_dependency)): dependency.some_method()  For the code snippet above, does subsequent visits to /example create a new instance of the MyDependency object each time? If so, how can I avoid that?

Answer: Yes, each request will receive a new instance. If you don't want that to happen, use a cache decorator, such as the built-in lru_cache in functools: - it's just a regular function, so any decorators will still be invoked (since they replace the original function with a new one which wraps the old one): from functools import lru_cache ... @lru_cache def get_dependency(): # Create and return an instance of the dependency return MyDependency()  However, if you use the same dependency multiple places in the hiearchy (for the same request), the same value will be re-used.  If one of your dependencies is declared multiple times for the same path operation, for example, multiple dependencies have a common sub-dependency, FastAPI will know to call that sub-dependency only once per request.
Question: Trying to convert a pl.Date column to UNIX epoch as is, without any timezone offset: import datetime import polars as pl df = pl.DataFrame( {'Date': [datetime.datetime.now().date()]} )  Correct time (00:00:00) when converted to Datetime: df.with_columns( pl.col("Date").cast(pl.Datetime) )  ┌─────────────────────┐ │ Date │ │ --- │ │ datetime[μs] │ ╞═════════════════════╡ │ 2023-06-10 00:00:00 │ └─────────────────────┘  Incorrect time when casting to timestamp: datetime.datetime.fromtimestamp( df.with_columns( pl.col("Date").cast(pl.Datetime).dt.timestamp("ms").truediv(1_000) ).item() )  datetime.datetime(2023, 6, 10, 8, 0) # (08:00:00)  As suggested, without casting to Datetime also produces the incorrect time. (08:00:00) pl.col("Date").dt.timestamp("ms").truediv(1_000) 

Answer: Note that vanilla Python datetime defaults to local time if you don't set a time zone (naive datetime). In contrast, polars assumes naive datetime to resemble UTC (as pandas does as well). Keep it consistent by setting the time zone, e.g. UTC: from datetime import datetime, timezone import polars as pl df = pl.DataFrame( {'Date': [datetime.now(timezone.utc).date()]} ) df = df.with_columns( pl.col("Date").cast(pl.Datetime).dt.timestamp("ms").truediv(1_000).alias("Unix") ) print(df) # shape: (1, 2) # ┌────────────┬──────────┐ # │ Date ┆ Unix │ # │ --- ┆ --- │ # │ date ┆ f64 │ # ╞════════════╪══════════╡ # │ 2023-06-10 ┆ 1.6864e9 │ # └────────────┴──────────┘ print(datetime.fromtimestamp(df["Unix"][0], timezone.utc)) # 2023-06-10 00:00:00+00:00
Question: What could be the reason for this error being thrown: Traceback (most recent call last): File "/Users/me/sc/sc.py", line 30, in <module> driver = Chrome(ChromeDriverManager().install(), options=chrome_options) TypeError: __init__() got multiple values for argument 'options' During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/Users/me/sc/sc.py", line 34, in <module> driver = Chrome("./chromedriver", options=chrome_options) TypeError: __init__() got multiple values for argument 'options' During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/Users/me/sc/sc.py", line 36, in <module> driver = Chrome("chromedriver.exe", options=chrome_options) TypeError: __init__() got multiple values for argument 'options' List item  For this code: from time import time, sleep from selenium.webdriver import Chrome from selenium.webdriver.chrome.options import Options from selenium.webdriver.common.by import By from selenium.webdriver.remote.webelement import WebElement from selenium.webdriver.support import expected_conditions as EC from selenium.webdriver.support.wait import WebDriverWait from webdriver_manager.chrome import ChromeDriverManager from selenium.webdriver.common.action_chains import ActionChains from args_parser import ArgsParser from downloader import download_file args = ArgsParser() def print_if_verbose(val): if args.output_verbose: print(val) WAITING_TIMEOUT = 180 chrome_options = Options() driver_user_agent = ('Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 ' '(KHTML, like Gecko) Chrome/90.0.4430.93 Safari/537.36') chrome_options.add_argument(f'user-agent={driver_user_agent}') if not args.display_browser: chrome_options.add_argument('--headless') try: driver = Chrome(ChromeDriverManager().install(), options=chrome_options) except Exception as e: print(e) try: driver = Chrome("./chromedriver", options=chrome_options) except Exception: driver = Chrome("chromedriver.exe", options=chrome_options)  BTW, I have Chrome 114 on macOS M2 silicon, and using Python 3.9

Answer: This is due to changes in selenium 4.10.0: https://github.com/SeleniumHQ/selenium/commit/9f5801c82fb3be3d5850707c46c3f8176e3ccd8e  Note that the first argument is no longer executable_path, but options. (That's why it complains that you're passing it in twice.) If you want to pass in an executable_path, you'll have to use the service arg now. Example: from selenium import webdriver from selenium.webdriver.chrome.service import Service service = Service(executable_path=r'./chromedriver') options = webdriver.ChromeOptions() options.add_argument('--headless') driver = webdriver.Chrome(service=service, options=options) # ... driver.quit()  Also note that a driver manager is now built-in to selenium, so you no longer need to use the separate webdriver_manager. The Selenium Team talked about that here: https://www.linkedin.com/pulse/selenium-manager-best-tool-from-you-can-forget-david-burns/
Question: I was making a personal assistant. I got an error in starting code: import pyttsx3 engine = pyttsx3.init() engine.say('How are you today?') engine.runAndWait()  Error: /usr/local/lib/python3.11/site-packages/pyttsx3/drivers/nsss.py:12: ObjCSuperWarning: Objective-C subclass uses super(), but super is not objc.super class NSSpeechDriver(NSObject): Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/pyttsx3/__init__.py", line 20, in init eng = _activeEngines[driverName] ~~~~~~~~~~~~~~^^^^^^^^^^^^ File "/usr/local/Cellar/python@3.11/3.11.3/Frameworks/Python.framework/Versions/3.11/lib/python3.11/weakref.py", line 136, in __getitem__ o = self.data[key]() ~~~~~~~~~^^^^^ KeyError: None During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/Users/anshtyagi/Documents/personal assistant/main.py", line 5, in <module> engine = pyttsx3.init() ^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/pyttsx3/__init__.py", line 22, in init eng = Engine(driverName, debug) ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/pyttsx3/engine.py", line 30, in __init__ self.proxy = driver.DriverProxy(weakref.proxy(self), driverName, debug) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/pyttsx3/driver.py", line 52, in __init__ self._driver = self._module.buildDriver(weakref.proxy(self)) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/pyttsx3/drivers/nsss.py", line 9, in buildDriver return NSSpeechDriver.alloc().initWithProxy(proxy) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/pyttsx3/drivers/nsss.py", line 15, in initWithProxy self = super(NSSpeechDriver, self).init() ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'super' object has no attribute 'init' sys:1: UninitializedDeallocWarning: leaking an uninitialized object of type NSSpeechDriver  I don't know what is the problem. One more thing: due to some issue I had to uninstall old python version on Mac and installed new one using Homebrew. Mac OS ventura 13.4 Python 3.11

Answer: this turns out to be a little tricky. and this is a workaround! hope works for you. Under the hood, this module pyttsx3 uses PyObjC as a bridge between Python and Objective-C. Step 1: Check that pyobjc is installed(pip show pyobjc), if not install as pip install pyobjc.  Step 2: open this file /usr/local/lib/python3.11/site-packages/pyttsx3/drivers/nsss.py and change the following: #self = super(NSSpeechDriver, self).init() comment this line , and add the following self = objc.super(NSSpeechDriver, self).init()   Note: from Foundation import * imports NSObject and objc from foundation, which has been consumed. after the change, your following program would run okay. import pyttsx3 engine = pyttsx3.init() engine.say('How are you today?') engine.runAndWait()
Question: I want to create a pandas dataframe from a given start and end date: import pandas as pd from pandas.tseries.offsets import MonthEnd start_date = "2020-05-17" end_date = "2020-07-23"  For each row in this dataframe, I should have the start day and end day of the month, so the expected output is: start end month year 2020-05-17 2020-05-31 May 2020 2020-06-01 2020-06-30 June 2020 2020-07-01 2020-07-23 July 2020  I know I have to loop over each month between the interval created by start_date and end_date. While I know how to extract the last day in a date: def last_day(date: str): return pd.Timestamp(date) + MonthEnd(1)  I'm stuck over how to run this over the interval. Any suggestion will be appreciated.

Answer: You can use pd.date_range and pd.to_datetime: start = pd.to_datetime([start_date] + pd.date_range(start_date, end_date, freq='MS').tolist()) end = pd.to_datetime(pd.date_range(start_date, end_date, freq='M').tolist() + [end_date]) month = start.strftime('%B') year = start.year df = pd.DataFrame({'start': start, 'end': end, 'month': month, 'year': year})  Output: >>> df start end month year 0 2020-05-17 2020-05-31 May 2020 1 2020-06-01 2020-06-30 June 2020 2 2020-07-01 2020-07-23 July 2020
Question: Given an array of integers, I would like to obtain an array of the same size where every value is a number of occurrences of a corresponding element in the original array. For example given the following array: a = np.array([1, 1, 4, 10, 5, 3, 5, 5, 8, 9])  This should be the result: array([2, 2, 1, 1, 3, 1, 3, 3, 1, 1])  Even though it is straightforward to achieve this via collections.Counter or built-in list.count(), I'm looking for a more performant way to work with large lists.

Answer: You could use np.unique and use the parameter return_inverse and return_counts. Use return_inverse to index return_counts to get desired results.  return_inverse bool, optional If True, also return the indices of the unique array (for the specified axis, if provided) that can be used to reconstruct ar. return_counts bool, optional If True, also return the number of times each unique item appears in ar.  _, idx, c = np.unique(a, return_inverse=True, return_counts=True) c[idx] # array([2, 2, 1, 1, 3, 1, 3, 3, 1, 1])
Question: I am making a heatmap in seaborn. I am using 'viridis', but I modify it slightly so some of the values get particular colors. In my MWE, .set_over is used to set the values above 90 to 'black', and .set_under is used to set the values below 10 to 'white'. I also mask out part of the heatmap. This all works fine. How can I also map a middle range value, 20, to 'orange', and without effecting the current colorbar appearance? As you can see, .set_over, and .set_under do not change the colorbar appearance. import matplotlib import seaborn as sns import numpy as np np.random.seed(7) A = np.random.randint(0,100, size=(20,20)) mask_array = np.zeros((20, 20), dtype=bool) mask_array[:, :5] = True cmap = matplotlib.colormaps["viridis"] # Set the under color to white cmap.set_under("white") # Set the voer color to white cmap.set_over("black") # Set the background color g = sns.heatmap(A, vmin=10, vmax=90, cmap=cmap, mask=mask_array) # Set color of masked region g.set_facecolor('lightgrey')   I have seen Map value to specific color in seaborn heatmap, but I am not sure how I can use it to solve my problem.

Answer: Pulling from this answer, here is a solution that uses a mask rather than a custom colorbar: import matplotlib import seaborn as sns import numpy as np from matplotlib.colors import ListedColormap np.random.seed(7) A = np.random.randint(0,100, size=(20,20)) mask_array = np.zeros((20, 20), dtype=bool) mask_array[:, :5] = True # cmap = matplotlib.colormaps["viridis"] cmap = matplotlib.cm.get_cmap('viridis') # Set the under color to white cmap.set_under("white") # Set the voer color to white cmap.set_over("black") # Set the background color g = sns.heatmap(A, vmin=10, vmax=90, cmap=cmap, mask=mask_array) # Set color of masked region g.set_facecolor('lightgrey') special_data = np.ma.masked_where(A==20, A) sns.heatmap(special_data, cmap=ListedColormap(['orange']), mask=(special_data != 1), cbar=False)