eshangj/stackoverflow_q_and_a_sample huggingface dataset:
Question: I have an array with shape (100, 80, 3) which is an rgb image. I have a boolean mask with shape (100, 80). I want each pixel where the mask is True to have value of pix_val = np.array([0.1, 0.2, 0.3]). cols = 100 rows = 80 img = np.random.rand(rows, cols, 3) mask = np.random.randint(2, size=(rows, cols), dtype=np.bool_) px = np.array([0.1, 0.2, 0.3]) for ch in range(3): img[:, :, ch][mask] = px[ch]  I thought broadcasting: img[mask[:, :, None]] = px  would work. But it did not. I am looking for a vectorized (efficient) way to implement it.

Answer: I'll attempt to explain why your indexing attempt didn't work. Make a smaller 3d array, and 2d mask: In [1]: import numpy as np In [2]: img = np.arange(24).reshape(2,3,4) In [3]: mask = np.array([[1,0,1],[0,1,1]],bool);mask Out[3]: array([[ True, False, True], [False, True, True]])  Using @mozway's indexing, produces a (4,4) array. The first first 4 is the number of True values in the mask, the second is the trailing dimension: In [4]: img[mask] Out[4]: array([[ 0, 1, 2, 3], [ 8, 9, 10, 11], [16, 17, 18, 19], [20, 21, 22, 23]])  With your indexing attempt, we get an error. (You really should have shown the error message): In [5]: img[mask[:,:,None]] --------------------------------------------------------------------------- IndexError Traceback (most recent call last) Cell In[5], line 1 ----> 1 img[mask[:,:,None]] IndexError: boolean index did not match indexed array along dimension 2; dimension is 4 but corresponding boolean dimension is 1  With this None, mask dimension is (2,3,1). That last 1 doesn't match the 4 of img. broadcasting doesn't apply in this context. Now if we attempt to use mask in a multiplication, the (2,3,4) and (2,3) don't broadcast together: In [6]: img*mask --------------------------------------------------------------------------- ValueError Traceback (most recent call last) Cell In[6], line 1 ----> 1 img*mask ValueError: operands could not be broadcast together with shapes (2,3,4) (2,3)  But (2,3,1) does broadcast with (2,3,4), producing a select number of 0 rows: In [7]: img*mask[:,:,None] Out[7]: array([[[ 0, 1, 2, 3], [ 0, 0, 0, 0], [ 8, 9, 10, 11]], [[ 0, 0, 0, 0], [16, 17, 18, 19], [20, 21, 22, 23]]])  As I commented, using a boolean mask is equivalent to indexing with nonzero arrays: In [13]: I,J = np.nonzero(mask); I,J Out[13]: (array([0, 0, 1, 1], dtype=int64), array([0, 2, 1, 2], dtype=int64)) In [14]: img[I,J,:] Out[14]: array([[ 0, 1, 2, 3], [ 8, 9, 10, 11], [16, 17, 18, 19], [20, 21, 22, 23]])  In the assignment expresion, a size (4,) value can broadcast to the (n,4) indexed img[mask]. Now if we were attempting to mask other dimensions we might need to make a px[:,None,:] or something like that.
Question: With pydantic, is there a way for mypy to be hinted so it doesn't raise an error in this scenario, where there's a field_validator modifying the type? class MyModel(BaseModel): x: int @field_validator("x", mode="before") @classmethod def to_int(cls, v: str) -> int: return len(v) MyModel(x='test') 

Answer: If you're always inserting a string there, it might be better to use a computed_field. Something along this, maybe? class MyModel(BaseModel): input: str @computed_field def x(self) -> int: return len(self.input)  I think it's very counterintuitive if you see the model with the int declaration while it would raise type errors if you put an integer inside a JSON at that place.
Question: I have a Flask service that receives GET requests, and I want to scale the QPS on that endpoint (on a single machine/container). Should I use a python ThreadPoolExecutor or ProcessPoolExecutor, or something else? The GET request just retrieves small pieces of data from a cache backed by a DB. Is there anything specific to Flask that should be taken into account?

Answer: Neither. Flask will serve one request per worker (or more, but depending on the worker type) - the way you set-it up, either with gunicorn, wsgi or awsgi is what is responsible for the number of parallel requests your app can process. Inside your app, you don't change anything - your views will be called as independent processes, independent threads or independent async tasks, depending on how you setup your server - that is where you have to tinker with the configurations. Using another concurrency strategy would only make sense if each request would have calculations and data fetching which could themselves be parallelized, inside the same request. Check how is your deployment configured, and pick the best option for you (all things being equal, pick the easiest one): https://flask.palletsprojects.com/en/stable/deploying/ (also, I would not recomend "mod_wsgi" among those options - it is super complex and old tech)
Question: Let's say I have DataFrames df1 and df2: >>> df1 = pd.DataFrame({'A': [0, 2, 4], 'B': [2, 17, 7], 'C': [4, 9, 11]}) >>> df1 A B C 0 0 2 4 1 2 17 9 2 4 7 11 >>> df2 = pd.DataFrame({'A': [9, 2, 32], 'B': [1, 3, 8], 'C': [6, 2, 41]}) >>> df2 A B C 0 9 1 6 1 2 3 2 2 32 8 41  What I want is the 3rd DataFrame that will have minimal rows (min is calculated based on column B), that is: >>> df3 A B C 0 9 1 6 1 2 3 2 2 4 7 11  I really don't want to do this by iterating over all rows and comparing them one by one, is there a faster and compact way to do this?

Answer: You can mask df1 with df2 when df2['B'] < df1['B']: out = df1.mask(df2['B'].lt(df1['B']), df2)  Output:  A B C 0 9 1 6 1 2 3 2 2 4 7 11
Question: I have a pandas DataFrame with a column like: 0 1 1 2 2 3 4 5 5 0 0 0  I would like to leave any leading zeros, but ffill to replace the trailing zeros with one greater than the previous, nonzero value. In this case, I'd like the output to be: 0 1 1 2 2 3 4 5 5 6 6 6  How can I go about doing this?

Answer: You could mask, increment and ffill: m = df['col'].eq(0) s = df['col'].mask(m) df['out'] = s.fillna(s.add(1).ffill().fillna(0)).convert_dtypes()  Or, if you really want to only target the trailing zeros: df['out'] = df['col'].mask(df['col'].eq(0)[::-1].cummin(), df['col'].max()+1)  Output:  col out 0 0 0 1 1 1 2 1 1 3 2 2 4 2 2 5 3 3 6 4 4 7 5 5 8 5 5 9 0 6 10 0 6 11 0 6  Intermediates (first approach):  col out m s s.add(1) .ffill() .fillna(0) 0 0 0 True NaN NaN NaN 0.0 1 1 1 False 1.0 2.0 2.0 2.0 2 1 1 False 1.0 2.0 2.0 2.0 3 2 2 False 2.0 3.0 3.0 3.0 4 2 2 False 2.0 3.0 3.0 3.0 5 3 3 False 3.0 4.0 4.0 4.0 6 4 4 False 4.0 5.0 5.0 5.0 7 5 5 False 5.0 6.0 6.0 6.0 8 5 5 False 5.0 6.0 6.0 6.0 9 0 6 True NaN NaN 6.0 6.0 10 0 6 True NaN NaN 6.0 6.0 11 0 6 True NaN NaN 6.0 6.0  Intermediates (second approach):  col out m s df['col'].eq(0) [::-1].cummin() 0 0 0 True NaN True False 1 1 1 False 1.0 False False 2 1 1 False 1.0 False False 3 2 2 False 2.0 False False 4 2 2 False 2.0 False False 5 3 3 False 3.0 False False 6 4 4 False 4.0 False False 7 5 5 False 5.0 False False 8 5 5 False 5.0 False False 9 0 6 True NaN True True 10 0 6 True NaN True True 11 0 6 True NaN True True  applying per group: Assuming a group LOT_ID and the target column STEP_NUMBER: df['out'] = (df.groupby('LOT_ID')['STEP_NUMBER'] .transform(lambda x: x.mask(x.eq(0)[::-1].cummin(), x.max()+1)) )
Question: I am using the Python Client to amend a Google Sheet shared to a Google Service Account. Everything works as expected when using a Sheet that is created by the Service Account, but returns a Requested entity was not found error when accessing a Google Sheet that is owned by someone else but shared with the service account. I have tried with all combinations of these definitions:  Personal Google account (i.e. @gmail.com) Workspace Google account Service Account with "shared outside domain set" with Drive and Sheets APIs enabled "Stock" Service account with Drive and Sheets APIs enabled Shared the containing folder of the Sheet Shared just the Sheet itself  Can anyone point to what I could be doing wrong that would mean that the client can find a sheet that it creates, but not one that is shared with it?

Answer:  Can anyone point to what I could be doing wrong that would mean that the client can find a sheet that it creates, but not one that is shared with it?  There is a scope in Google drive, which is characteristic of your description: https://www.googleapis.com/auth/drive.file See, edit, create, and delete only the specific Google Drive files you use with this app  If you're using that scope, change to a more permissive scope like https://www.googleapis.com/auth/drive
Question: What is the preferred way to assign/add a new column to a polars dataframe in .select() or .with_columns()? Are there any differences between the below column assignments using .alias() or the = sign? import polars as pl df = pl.DataFrame({"A": [1, 2, 3], "B": [1, 1, 7]}) df = df.with_columns(pl.col("A").sum().alias("a_sum"), another_sum=pl.col("A").sum() )  I am not sure which one to use.

Answer: The advantage of alias is that it allows you to specify a column name that wouldn't be a valid Python identifier. For example, you could use "a sum!". This can also be achieved by creating a dictionary and using ** to unpack it, passing the items as keyword arguments. Assignment with = cannot work in this way, as it requires a valid identifier (e.g., another_sum). df = df.with_columns(pl.col("A").sum().alias("a sum!"), another_sum=pl.col("A").sum(), **{":) \u2014 also a sum": pl.col("A").sum()} )  Output: shape: (3, 5) ┌─────┬─────┬────────┬─────────────┬─────────────────┐ │ A ┆ B ┆ a sum! ┆ another_sum ┆ :) — also a sum │ │ --- ┆ --- ┆ --- ┆ --- ┆ --- │ │ i64 ┆ i64 ┆ i64 ┆ i64 ┆ i64 │ ╞═════╪═════╪════════╪═════════════╪═════════════════╡ │ 1 ┆ 1 ┆ 6 ┆ 6 ┆ 6 │ │ 2 ┆ 1 ┆ 6 ┆ 6 ┆ 6 │ │ 3 ┆ 7 ┆ 6 ┆ 6 ┆ 6 │ └─────┴─────┴────────┴─────────────┴─────────────────┘
Question: I'm trying to plot 3 planes in 3D space using plotly, I can only define a surface along the XY plane, whilst ZY and XZ do not appear. I'm including a simple example below, I would expect the code to produce three planes intersecting at the point (1, 1, 1), instead there is only one surface at (x, y) = 1. Any help would be greatly appreciated. import plotly.graph_objects as go import numpy as np zsurf = go.Surface(y=[0, 1, 2], x=[0, 1, 2], z=np.ones((3, 3))) ysurf = go.Surface(x=[0, 1, 2], z=[0, 1, 2], y=np.ones((3, 3))) xsurf = go.Surface(z=[0, 1, 2], y=[0, 1, 2], x=np.ones((3, 3))) fig = go.Figure() fig.add_trace(zsurf) fig.add_trace(ysurf) fig.add_trace(xsurf) fig.show() 

Answer: All of your x, y, z should have the same shape. I don't because of which default UB one of your plane even show. But none of them are correctly specified. What you meant is import plotly.graph_objects as go import numpy as np xx,yy=np.meshgrid([0,1,2], [0,1,2]) zsurf = go.Surface(y=xx, x=yy, z=np.ones((3, 3))) ysurf = go.Surface(x=xx, z=yy, y=np.ones((3, 3))) xsurf = go.Surface(z=xx, y=yy, x=np.ones((3, 3))) fig = go.Figure() fig.add_trace(zsurf) fig.add_trace(ysurf) fig.add_trace(xsurf) fig.show()   A surface (from plotly Surface point of view) is a 2D mesh of points in space. So a 2D array of points. That is 3 2D arrays, one for x, one for y, one for z. Each point being located in space Or, to be more explicit (it is exactly the same as my first answer. Just, the mesh looks less magic when written explicitly rather that with meshgrid) import plotly.graph_objects as go import numpy as np zsurf = go.Surface(y=[[0,1,2],[0,1,2],[0,1,2]], x=[[0,0,0],[1,1,1],[2,2,2]], z=np.ones((3, 3))) ysurf = go.Surface(x=[[0,1,2],[0,1,2],[0,1,2]], z=[[0,0,0],[1,1,1],[2,2,2]], y=np.ones((3, 3))) xsurf = go.Surface(z=[[0,1,2],[0,1,2],[0,1,2]], y=[[0,0,0],[1,1,1],[2,2,2]], x=np.ones((3, 3))) fig = go.Figure() fig.add_trace(zsurf) fig.add_trace(ysurf) fig.add_trace(xsurf) fig.show()
Question: I am using Django to serve an application, and I noticed some slowing down recently. So I went and checked the console that serves the server, that usually logs lines of this format : <date_time> "GET <path> HTTP/1.1" <HTTP_STATUS> <response_time> What I thought was the response time in milliseconds is apparently not, as I get values that would be ludicrous (example 3923437 for a query that when timed in python directly takes 0.936 seconds). I'm pretty sure it's a response time though, as it's always scaled with the time I wait. Can someone explain to me what that number is ? I couldn't find where this default log is documented.

Answer: It's the response size and it's printed by Python http-server which Django inherits. That explains why it's not documented by Django, because it's not the Django code that prints it. You can verify that by looking at this Django module. This is the line that starts the http-server. It inherits from Python http-server. This is the line that prints the response size.
Question: Suppose we have a project structure like: project/ public_app/ __init__.py dir/ __init__.py config.py subdir/ __init__.py functions.py utils.py my_app/ main.py  In my_app/main.py, I would like to import some functions from public_app/dir/subdir/functions.py. A solution I found was to add the following: # main.py import sys import os path = os.path.abspath(os.path.join(os.path.dirname(__file__), '../')) sys.path.append(path) from public_app.dir.subdir.functions import *  This seems to work, except now I would also like to import from public_app/dir/subdir/utils.py. However inside this file, it contains other relative imports: # utils.py from dir.config import *  If I then try doing # main.py from public_app.dir.subdir.utils import *  this gives me a ModuleNotFoundError: No module named 'dir'. Any suggestion on how to do this? Note that I would ideally like to not mess with public_app at all. This is because it is a frequently updated directory pulled from a public repository, and would require constantly changing the imports. I would also like to also keep my_app in a separate directory for cleanliness/easier maintenance if possible.  Edit: Figured it out actually by sheer chance. See below for answer.

Answer: Run the main.py from the parent directory as a reference. That will resolve relative errors and import errors as the interpreter will get the idea of the relations between modules. Thus, Instead of doing: .../myapp$ python3 main.py  Do: <parentfolderofproject>$ python3 -m project.myapp.main
Question: I have used django ModelAdmin with M2M relationship and formfield filtering code as follows: But for superuser or any other login where the number of mailboxes is more than 100k. I have sliced the available after filtering. But loading the m2m field takes time and times out for superuser login: def formfield_for_manytomany(self, db_field, request, **kwargs): if db_field.name == "mailboxes": if request.user.is_superuser: queryset = Mailbox.objects.prefetch_related('domain').only('id','email') kwargs["queryset"] = queryset field = super().formfield_for_manytomany(db_field, request, **kwargs) field.widget.choices.queryset = queryset # Limit visible options return field if request.user.groups.filter(name__in=['customers']).exists(): queryset = Mailbox.objects.filter( domain__customer__email=request.user.email).prefetch_related('domain').only('id','email') kwargs["queryset"] = queryset field = super().formfield_for_manytomany(db_field, request, **kwargs) field.widget.choices.queryset = queryset return field return super().formfield_for_manytomany(db_field, request, **kwargs)  I want to use filter_horizontal only and not django auto_complete_light or any javascript. how can the latency be resolved. As you can see the queryset filtering is already done to get valid options. Slicing removed the mailbox model is simple: class Mailbox(AbstractPerson): username = models.EmailField(verbose_name='email', blank=True) email = models.EmailField(verbose_name='email', null=True,blank=True, unique=True) local_part = models.CharField(max_length=100,verbose_name='user part',help_text=hlocal_part) domain = models.ForeignKey(Domain, on_delete=models.CASCADE)  which has M2M relation with GroupMailIds model: class GroupMailIds(models.Model): local_part = models.CharField(max_length=100,verbose_name='local part',help_text=hlocal_part) address = models.EmailField(unique=True,verbose_name='Email id of the distribution list') domain = models.ForeignKey(Domain, on_delete=models.CASCADE,related_name='domains') mailboxes = models.ManyToManyField(Mailbox,related_name='my_mailboxes') 

Answer: Limited the number of objects loading during form init as: class GroupMailIdsForm(forms.ModelForm): class Meta: model = GroupMailIds fields = "__all__" def __init__(self, *args, **kwargs): super().__init__(*args, **kwargs) request = getattr(self, 'request', None) if not request: return email = request.user.email qs = Mailbox.objects.all() if request.user.is_superuser: filtered_qs = qs else: filtered_qs = qs.none() # No access if none of the conditions match # Set the filtered queryset to the form field self.fields['mailboxes'].queryset = filtered_qs  In ModelAdmin: def get_form(self, request, obj=None, **kwargs): # Pass the request to the form form = super().get_form(request, obj, **kwargs) form.request = request # Attach the request to the form return form  The init for non-superuser initializes the queryset with None and during the admin, populates the dropdown with required filtered values during admin based on login value
Question: I'm trying to pre-generate and store matplotlib figures in python, and then display them on a keyboard event (left-right cursor keys). It partially seems working, but fails after the first keypress. Any idea, what am I doing wrong? import matplotlib.pyplot as plt import numpy as np def new_figure(title, data): fig,ax = plt.subplots() plt.plot(data, label=title) ax.set_xlabel('x-axis') ax.set_ylabel('value') plt.legend() plt.title(title) plt.close(fig) return fig def show_figure(fig): dummy = plt.figure() new_manager = dummy.canvas.manager new_manager.canvas.figure = fig fig.set_canvas(new_manager.canvas) def redraw(event, cnt): event.canvas.figure.clear() dummy = event.canvas.figure new_manager = dummy.canvas.manager new_manager.canvas.figure = figs[cnt] figs[cnt].set_canvas(new_manager.canvas) event.canvas.draw() def keypress(event): global cnt if event.key == 'right': cnt += 1 cnt %= mx elif event.key == 'left': cnt -= 1 if cnt < 0: cnt = mx-1 redraw(event, cnt) d = range(0, 360) data = [] data.append(np.sin(np.radians(d))) data.append(np.cos(np.radians(d))) data.append(np.tan(np.radians(d))) titles = ['sin','cos','tan'] mx = len(data) figs = [] for i in range(mx): fig = new_figure(titles[i], data[i]) figs.append(fig) cnt = 0 show_figure(figs[0]) figs[0].canvas.mpl_connect('key_press_event', keypress) plt.show()  The error I get eventually is:  File "C:\Program Files\Python39\lib\tkinter\__init__.py", line 1636, in _configure self.tk.call(_flatten((self._w, cmd)) + self._options(cnf)) _tkinter.TclError: invalid command name ".!navigationtoolbar2tk.!button2" 

Answer: Not sure about the root cause of the error, but one way to avoid this is to fully replace the figure and reconnect the key_press_event: def redraw(event, cnt): event.canvas.figure = figs[cnt] event.canvas.mpl_connect('key_press_event', keypress) event.canvas.draw()
Question: I'm trying to sum up data of selected date, month of selected date and previous month of selected date but don't know how to do. Below is my sample data and my expected Output: Sample data: import pandas as pd import numpy as np df = pd.read_excel('https://github.com/hoatranobita/hoatranobita/raw/refs/heads/main/Check%20data%20(1).xlsx', sheet_name='Data') df COA Code USDConversion Amount Base Date 2 0 19010000000 26924582.44 2024-10-01 1 19010000000 38835600.44 2024-10-02 2 19010000000 46794586.57 2024-10-03 3 19010000000 57117346.49 2024-10-06 4 19010000000 69256132.98 2024-10-07 ... ... ... ... 65 58000000000 38082130.88 2024-11-12 66 58000000000 38140016.13 2024-11-13 67 58000000000 38160089.27 2024-11-14 68 58000000000 38233974.54 2024-11-17 69 58000000000 38323598.99 2024-11-18  So if I select date of November (for example 2024-11-18, I want to group by selected date, month of selected date and previous month of selected date. Output: COA Code 2024-11-18 October November 0 19010000000 42625047.24 1354513618.61 584813860.97 1 58000000000 38323598.99 820927014.08 456265522.64 

Answer: The exact generalization of your question is not fully clear, but assuming that you want to group by COA Code, you could ensure everything is a datetime/periods, then select the appropriate rows with boolean indexing and between, finally perform a groupby.sum of those rows and concat to the original date rows. Here as a function for clarity: def get_previous(df, date, date_col='Base Date 2'): # ensure working with datetime/period objects date = pd.Timestamp(date) period = date.to_period('M') dt = pd.to_datetime(df[date_col]) p = dt.dt.to_period('M') # select rows to keep m = p.between(period-1, period, inclusive='both') # produce rows with original date # aggregate previous and current month # combine and rename the columns return pd.concat([ df[df[date_col].eq(date)] .set_index('COA Code')['USDConversion Amount'] .rename(date.strftime('%Y-%m-%d')), df[m].groupby(['COA Code', p])['USDConversion Amount'] .sum().unstack(date_col) .rename(columns=lambda x: x.strftime('%B')) ], axis=1).reset_index() out = get_previous(df, '2024-11-18')  Output:  COA Code 2024-11-18 October November 0 58000000000 38323598.99 8.209270e+08 4.562655e+08  NB. you can replace groupby.sum+unstack by pivot_table (df[m].assign(col=p).pivot_table(index='COA Code', columns='col', values='USDConversion Amount', aggfunc='sum')).
Question: To search over multiple columns, and create a new column of flag if string found, the following codes work, but is there any compact way inside with_columns() to achieve the same? df = pl.DataFrame({ "col1": ["hello", "world", "polars"], "col2": ["data", "science", "hello"], "col3": ["test", "string", "match"], "col4": ["hello", "example", "test"] }) search_string = "hello" condition = pl.lit(False) for col in df.columns: condition |= pl.col(col).str.contains(search_string) df = df.with_columns( condition.alias("string_found") + 0 ) print(df)  shape: (3, 5) ┌────────┬─────────┬────────┬─────────┬──────────────┐ │ col1 ┆ col2 ┆ col3 ┆ col4 ┆ string_found │ │ --- ┆ --- ┆ --- ┆ --- ┆ --- │ │ str ┆ str ┆ str ┆ str ┆ i32 │ ╞════════╪═════════╪════════╪═════════╪══════════════╡ │ hello ┆ data ┆ test ┆ hello ┆ 1 │ │ world ┆ science ┆ string ┆ example ┆ 0 │ │ polars ┆ hello ┆ match ┆ test ┆ 1 │ └────────┴─────────┴────────┴─────────┴──────────────┘ 

Answer: You can use .any_horizontal() df.with_columns( pl.any_horizontal(pl.all().str.contains(search_string)) .alias("string_found") )  shape: (3, 5) ┌────────┬─────────┬────────┬─────────┬──────────────┐ │ col1 ┆ col2 ┆ col3 ┆ col4 ┆ string_found │ │ --- ┆ --- ┆ --- ┆ --- ┆ --- │ │ str ┆ str ┆ str ┆ str ┆ bool │ ╞════════╪═════════╪════════╪═════════╪══════════════╡ │ hello ┆ data ┆ test ┆ hello ┆ true │ │ world ┆ science ┆ string ┆ example ┆ false │ │ polars ┆ hello ┆ match ┆ test ┆ true │ └────────┴─────────┴────────┴─────────┴──────────────┘  You can replace pl.all() with pl.col(pl.String) to limit the expression to String columns only. In this example you only have String columns so it doesn't come into play.