eshangj/stackoverflow_q_and_a_sample huggingface dataset:
Question: Using python I would like to calculate all combinations of 3 from a list. For example, list = [a,b,c,d] and combinations would be - [a,b,c], [a,b,d], [a,c,d], [b,c,d]. And then I would like to add some items to the original list and get only the additional combinations of 3. For example, adding items [e,f] would generate new combinations - [a,b,e], [a,b,f], [a,c,e], [a,c,f], [a,d,e], [a,d,f], [a,e,f], [b,c,e], [b,c,f],... The lists will be large so we need to avoid generating the combinations twice and then filtering in order to get the 'additional combinations'. Background: I use itertools.combinations to get all combinations (of 3) for a list right now of about 100 items. That generates a lot of combinations and I doing a bunch of calculations and whatnot based on those combinations, looking for patterns and matches and stuff. I get through all that processing and if I don't have a 'successful' combination of 3 then I generate more candidates for the list (which in itself takes a long time). When I add the additional candidates to the list (usually like 10 or so), I then restart the analysis on the combinations which seems wasteful, so I would like to only be checking the 'additional' combinations.

Answer: Add the new items one by one: from itertools import combinations old = ['a', 'b', 'c', 'd'] new = ['e', 'f'] for item in new: for comb in combinations(old, 2): print(*comb, item) old.append(item)  Output (Attempt This Online!): a b e a c e a d e b c e b d e c d e a b f a c f a d f a e f b c f b d f b e f c d f c e f d e f  Or if order within each combination doesn't matter (inspired by a comment): from math import comb from itertools import combinations, islice old = ['a', 'b', 'c', 'd'] new = ['e', 'f'] new_combs = comb(len(old) + len(new), 3) - comb(len(old), 3) for comb in islice(combinations(new + old, 3), new_combs): print(*comb)  Output (Attempt This Online!): e f a e f b e f c e f d e a b e a c e a d e b c e b d e c d f a b f a c f a d f b c f b d f c d
Question: If I have a set of points: points = np.random.randint(0, 11, size=10) print(points)  Output: [ 5 4 9 7 4 1 2 10 4 2]  And if I want to get a matrix representing the distance from each point to each other point, I can do so like this: def get_diff_array(values): dX = values - values[0] tmp = [dX] for i in range(1, len(dX)): tmp.append((dX - dX[i])) return np.array(tmp) print(get_diff_array(points))  Output: [[ 0 -1 4 2 -1 -4 -3 5 -1 -3] [ 1 0 5 3 0 -3 -2 6 0 -2] [-4 -5 0 -2 -5 -8 -7 1 -5 -7] [-2 -3 2 0 -3 -6 -5 3 -3 -5] [ 1 0 5 3 0 -3 -2 6 0 -2] [ 4 3 8 6 3 0 1 9 3 1] [ 3 2 7 5 2 -1 0 8 2 0] [-5 -6 -1 -3 -6 -9 -8 0 -6 -8] [ 1 0 5 3 0 -3 -2 6 0 -2] [ 3 2 7 5 2 -1 0 8 2 0]]  Is there a faster NumPy-specific way to calculate this? Currently it takes approximately 0.44 seconds for 10k points, which seems slow. This is really just a learning question to try to understand NumPy better.

Answer: You can use values[:, np.newaxis] - values[np.newaxis, :]: import numpy as np def diff(A): return A[:, np.newaxis] - A[np.newaxis, :] print(diff(np.random.randint(0, 11, size=10)))  Prints [[ 0 -7 -8 -3 -6 -6 -4 -5 -10 -3] [ 7 0 -1 4 1 1 3 2 -3 4] [ 8 1 0 5 2 2 4 3 -2 5] [ 3 -4 -5 0 -3 -3 -1 -2 -7 0] [ 6 -1 -2 3 0 0 2 1 -4 3] [ 6 -1 -2 3 0 0 2 1 -4 3] [ 4 -3 -4 1 -2 -2 0 -1 -6 1] [ 5 -2 -3 2 -1 -1 1 0 -5 2] [ 10 3 2 7 4 4 6 5 0 7] [ 3 -4 -5 0 -3 -3 -1 -2 -7 0]]
Question: I have a DataFrame df with one time series variable, call it X. I can find the rolling mean over the last (say) n observations with df['rolling_mean'] = df.X.rolling(window=n).mean()  Now I want a new column with a boolean value that applies the following logic to each row and returns true if either:  X > rolling_mean and sometime within the next k observations, X attains a value less than rolling_mean or X < rolling_mean and sometime within the next k observations, X attains a value greater than rolling_mean.  My current workaround looks something like: new_column = [] for i, row in df.iterrows(): next_k = df[i:i + k] if row.X < row.rolling_mean and any(next_k.X > row.rolling_mean): new_column.append(1) elif row.X > row.rolling_mean and any(next_k.X < row.rolling_mean): new_column.append(1) else: new_column.append(0) df['new_column'] = new_column  But obviously this is iterative and not fast enough for a large dataset. Is there a fast/vectorized way of doing this?

Answer: A nice solution I came up with: df['rolling_max'] = df.X.rolling(k, min_periods=0).max().shift(-k) df['rolling_min'] = df.X.rolling(k, min_periods=0).min().shift(-k) df['will_drop_below_mean'] = (df.X > df.rolling_mean) & (df.rolling_min < df.rolling_mean) df['will_rise_above_mean'] = (df.X < df.rolling_mean) & (df.rolling_max > df.rolling_mean) df['new_column'] = ((df._will_drop_below_mean) | (df._will_rise_above_mean)).astype(int)  Basically using rolling().max() and rolling().min() with .shift() to create two boolean masks.
Question: I have a 3D array filled with RGB values and I need to replace some "pixels" with another value as fast as possible. The array looks like this: [[[ 78 77 75] [ 72 70 67] [ 72 70 67] ... [ 73 74 73] [ 71 72 71] [ 66 67 67]]]  I used numpy select to replace some values but this is not what I need as it doesn't check for the whole pixel RGB. np.select([array > 77, array < 77, [0, 0], array)  I tried to compare the whole RGB array with another one but it didn't work. np.select([array != [72, 70, 77]], [[0, 0, 0]], array)  Creating a mask didn't help either. I wasn't able to replace the mask with an RGB array. color = [30, 30, 57] mask = np.any(array != color, axis=2) 

Answer: You can define your two conditions, combine it using logical_or(), and then use where(): import numpy as np array = np.array([[[78, 77, 75], [72, 70, 67], [72, 70, 67]], [[78, 77, 75], [72, 70, 67], [72, 70, 67]], [[73, 74, 73], [71, 72, 71], [66, 67, 67]]]) color = [30, 30, 57] cond_a = array[:, :, 0] > 77 cond_b = array[:, :, 1] < 77 conds = np.logical_or(cond_a, cond_b) res = np.where(conds, color, array) print(res)  Prints  [[[30 30 57] [30 30 57] [30 30 57]] [[30 30 57] [30 30 57] [30 30 57]] [[30 30 57] [30 30 57] [30 30 57]]]
Question: I am trying to remove rows based on if the previous row was "added" and the current row is "removed". My goal is to identify the rows that are only "added". import pandas as pd df = pd.DataFrame({'Event_Time': ['2024-05-28 12:37:00', '2024-05-28 12:41:00', '2024-05-28 16:12:00','2024-05-08 09:40:00'], 'Name': ['Kenzie Doe', 'Kenzie Doe', 'Kenzie Doe', 'Abby Stone'], 'Action': ['Added', 'Removed', 'Added', 'Added']})  I converted the datetime so it can sort properly. The names are also sorted so all the actions for one user are together and by proper date time. I am not sure if muli-row and column actions can be taken in python. I would like my final result to look like this:    Event_time Name Action     2024-05-28 16:12:00 Kenzie Doe Added   2024-05-08 09:40:00 Abby Stone Added   

Answer: You can use a mask: ~(same & (rem | rem_add)): import pandas as pd df = pd.DataFrame({'Event_Time': ['2024-05-28 12:37:00', '2024-05-28 12:41:00', '2024-05-28 16:12:00', '2024-05-08 09:40:00'], 'Name': ['Kenzie Doe', 'Kenzie Doe', 'Kenzie Doe', 'Abby Stone'], 'Action': ['Added', 'Removed', 'Added', 'Added']}) df['Event_Time'] = pd.to_datetime(df['Event_Time']) df = df.sort_values(by=['Name', 'Event_Time']) same = df['Name'] == df['Name'].shift(-1) rem = (df['Action'] == 'Added') & (df['Action'].shift(-1) == 'Removed') rem_add = (df['Action'] == 'Removed') & (df['Action'].shift(-1) == 'Added') print(df[~(same & (rem | rem_add))])  Prints  Event_Time Name Action 3 2024-05-08 09:40:00 Abby Stone Added 2 2024-05-28 16:12:00 Kenzie Doe Added  Mask:  bitwise ~: inverse of the mask.  same: checks if the current row and the next row have the same name.  rem: checks if the current row has the action "Added" and the next row has the action "Removed".  rem_add: checks if the current row has the action "Removed" and the next row has the action "Added".  bitwise |: OR.  bitwise &: AND.
Question: I'm working on a webserver built with FastAPI version 0.111.0 and SQLModel version 0.0.18. When I'm calling the POST endpoint with partial data or with keys I don't specify in my class, it doesn't trigger the unprocessable entity or bad request error. I'm using the method POST only for inserting data and not updating it. For update an entity, I use a separate PUT request. Here's my model: from sqlmodel import SQLModel, Field class Job(SQLModel, table=True): job_id: int = Field(primary_key=True) title: str start_time: str end_time: str | None = None pipeline_id: int = Field(foreign_key="pipeline.pipeline_id") prj_path: str branch_tag: str user: str status: str log: str = "" result: str = ""  and here's the controller: from fastapi import APIRouter from models.job import Job job_router = APIRouter(prefix="/job", tags=["Jobs"]) @job_router.post("/", status_code=201, response_model=Job) async def add_job(job: Job): job = await c.insert_job(job) return job  The c.insert_job(job) is where I'm saving the object in the database and the job_router is importend in the main file with the FastAPI app: from fastapi import FastAPI from routers.job import job_router app = FastAPI() app.include_router(job_router)  Even if the auto-created documentation tells me that there are some required fields as shown here:  job class in swagger  if I'm sending a request with a body like { "job_id": 0 } or even { "job_id": 0, "test": "test" }, it passes without triggering anything like shown in the next image (here I return the partial object I get instead of adding it to the database because it would throw an error on fields without the default value):  returned object

Answer: Use this model for response validation  from pydantic import BaseModel from typing import Optional class JobRead(BaseModel): job_id: int title: str start_time: str end_time: Optional[str] = Field(default=None) pipeline_id: int prj_path: str branch_tag: str user: str status: str log: Optional[str] = Field(default="") result: Optional[str] = Field(default="")  And you can do use Pydantic validation about validation https://docs.pydantic.dev/latest/#why-use-pydantic or use Depends() with Pydantic model from fastapi import APIRouter from models.job import Job job_router = APIRouter(prefix="/job", tags=["Jobs"]) @job_router.post("/", status_code=201, response_model=Job) async def add_job(job: Job = Depends()): job = await c.insert_job(job) return job  about Depends https://fastapi.tiangolo.com/tutorial/dependencies/
Question: I have a class that holds an unspecified number of dicts as variables. I want to provide methods that allow the user to easily append to any of the dicts while naming the corresponding variable, without having to check whether the variable was already created. Example for my goal: >>> c = Container() >>> c.add_value(variable="first_dict", key="foo", value=1) >>> c.add_value(variable="first_dict", key="bar", value=2) >>> c.add_value(variable="second_dict", key="foo", value=2) >>> print(c.first_dict) {"foo":1, "bar":2) >>> print(c.second_dict) {"foo":2)  Currently, this is my solution: class Container(): def __init__(self): pass def add_value(self, variable: str, key: Any, value: Any): x = getattr(self, variable, {}) x[key] = value setattr(self, variable, x)  My concern is that accessing the attribute via getattr, then mutating it and setting it back via setattr introduces overhead that is not necessary. Is there a better way to write the Container.add_value()-method? Should I approach the whole problem differently?

Answer: You can initialize and access each sub-dict with the setdefault method of the object's attribute dict: class Container: def add_value(self, variable, key, value): vars(self).setdefault(variable, {})[key] = value
Question: I have a dataframe with an index of (month, A, B):  foo N month A B 1983-03-01 3 9 0 1 1983-06-01 3 9 0 1 1983-09-01 3 9 0 1 1983-11-01 4 5 0 1 1984-05-01 4 5 0 1 1984-06-01 3 9 0 1 1984-09-01 3 9 0 2  I would like to fill all missing dates, provided that a certain (A, B) combination exists in the index. What I do not want to do is to fill in the index for all (A, B) combinations. That is, I would like to have for (A=3, B=9) and for (A=4, B=5) month-indices running from 1983-03-01 to 1984-09-01 and 0s for filling. But I don't want there to be any records of (A=3, B=5) or (A=4, B=9). If this was a single index, I could simply idx = pd.date_range(df['month'].min(), df['month'].max(), freq='M') df = df.set_index('month') df.index = df.reindex(idx, fill_value=0)  How would I approach it in this situation? Worth noting that this solution should scale with a large number of unique values for A, B.

Answer: Assuming: df = pd.DataFrame({'month': pd.to_datetime(['1983-03-01', '1983-06-01', '1983-09-01', '1983-11-01', '1984-05-01', '1984-06-01', '1984-09-01']), 'A': [3, 3, 3, 4, 4, 3, 3], 'B': [9, 9, 9, 5, 5, 9, 9], 'foo': [0, 0, 0, 0, 0, 0, 0], 'N': [1, 1, 1, 1, 1, 1, 2]} )  You could use a groupby.apply: cols = df.columns.difference(['month', 'A', 'B']) out = (df.set_index('month').groupby(['A', 'B'])[cols] .apply(lambda x: x.reindex(pd.date_range(x.index.get_level_values('month').min(), x.index.get_level_values('month').max(), freq='MS').rename('month'), fill_value=0)) .reset_index()[df.columns] )  Output:  month A B foo N 0 1983-03-01 3 9 0 1 1 1983-04-01 3 9 0 0 2 1983-05-01 3 9 0 0 3 1983-06-01 3 9 0 1 4 1983-07-01 3 9 0 0 5 1983-08-01 3 9 0 0 6 1983-09-01 3 9 0 1 7 1983-10-01 3 9 0 0 8 1983-11-01 3 9 0 0 9 1983-12-01 3 9 0 0 10 1984-01-01 3 9 0 0 11 1984-02-01 3 9 0 0 12 1984-03-01 3 9 0 0 13 1984-04-01 3 9 0 0 14 1984-05-01 3 9 0 0 15 1984-06-01 3 9 0 1 16 1984-07-01 3 9 0 0 17 1984-08-01 3 9 0 0 18 1984-09-01 3 9 0 2 19 1983-11-01 4 5 0 1 20 1983-12-01 4 5 0 0 21 1984-01-01 4 5 0 0 22 1984-02-01 4 5 0 0 23 1984-03-01 4 5 0 0 24 1984-04-01 4 5 0 0 25 1984-05-01 4 5 0 1
Question: How to check are the attributes mutually exclusive? That is, can a word have more than one non-zero entry in the sentiment attributes? Would like to check if each attribute is mutually exclusive of one another and to enumerate through all pairs to compare each. N | P | U | L | S | W 0 1 ... 0 2 ... 1 0 ... 0 0 ... N P, N U, N L, N S, N W P U, P L, P S, P W U L, U S, U W L S, S W  Stumped using iloc here and if I can set my for loops to something like: a=range(0,7) b=range(0,7) for i in a: for j in b: sum(df.iloc[:,i]*df.iloc[:,j]!=0)  However I know this produces an error, how to then dynamically update the index within iloc to iterate through each column? If the sum is 0, then col 0 and 1 are mutually exclusive. Now to iterate over all columns and enumerate through nested loops.

Answer:  Check all pairs. The pair is mutually exclusive, if (df[ii] * df[jj] != 0).sum() == 0:  import pandas as pd def get_mut_ex(D): df = pd.DataFrame(D) cols = df.columns res, n = [], len(cols) for i in range(n): for j in range(i + 1, n): ii, jj = cols[i], cols[j] print(df[jj], df[ii]) if (df[ii] * df[jj] != 0).sum() == 0: res += [(ii, jj)] return res D = { 'N': [0, 0, 1, 0], 'P': [1, 2, 0, 0], 'U': [0, 0, 0, 0], 'L': [0, 0, 0, 0], 'S': [0, 0, 0, 0], 'W': [0, 0, 0, 0] } print(get_mut_ex(D))  Prints [('N', 'P'), ('N', 'U'), ('N', 'L'), ('N', 'S'), ('N', 'W'), ('P', 'U'), ('P', 'L'), ('P', 'S'), ('P', 'W'), ('U', 'L'), ('U', 'S'), ('U', 'W'), ('L', 'S'), ('L', 'W'), ('S', 'W')]
Question: I am trying to create a new columns of percentages which is a product of value in a row above and the value in the same row of another column. I have tried using shift and loc with no luck. I have tried using: df.at[0,'new_col'] = df.at[0,'other_col'] This first part works well and then df['new_col'] = df['new_col'].shift(1)*df['other_col'] this however does not work. My data example is as follows:    time val adj_val     0 1 1   1 0.5 0.5   2 0.6 0.3   3 0.7 0.21   4 0.9 0.189    I have been trying to work around this for a well such as using df.loc but with no luck. The values in adj_col are calculated as follows: 1 = 1 as per the first line of code - this works then 0.5 = 1 * 0.5 - the 1 is the first value in adj_val and 0.5 is in the val colum 0.21 = 0.3*0.7 0.189 = 0.9*0.21 ``` 

Answer: You cannot achieve the desired result with shift. shift will give you access to the previous row before any computation is performed, but you need the new value to become the next reference. What you want is a cumulative product, there is already a method for that in pandas: cumprod: df['adj_val'] = df['val'].cumprod()  It is also achievable with numpy.multiply.accumulate: import numpy as np df['adj_val'] = np.multiply.accumulate(df['val'])  Output:  time val adj_val 0 0 1.0 1.000 1 1 0.5 0.500 2 2 0.6 0.300 3 3 0.7 0.210 4 4 0.9 0.189
Question: I have the following pandas dataframe:    key1 key2 col_name bool col_1 col_2 col_3     a1 a2 col_1 0 5 10 20   b1 b2 col_3 1 10 10 5   c1 c2 col_1 1 5 15 5    Where bool==1, I would like to update the value in the column given by the col_name column to be 100. Expected output:    key1 key2 col_name bool col_1 col_2 col_3     a1 a2 col_1 0 5 10 20   b1 b2 col_3 1 10 10 100   c1 c2 col_1 1 100 15 5    I can do this by iterating through the table, but from what I've read this is never best practice. What would be the most efficient way of doing this?

Answer: Build a boolean mask with numpy and update: # identify cells for which the col_name matches the column name # only keep those that have a bool of 1 in the row m = ((df['col_name'].to_numpy()[:, None] == df.columns.to_numpy()) & df['bool'].eq(1).to_numpy()[:, None] ) df[m] = 100  Output:  key1 key2 col_name bool col_1 col_2 col_3 0 a1 a2 col_1 0 5 10 20 1 b1 b2 col_3 1 10 10 100 2 c1 c2 col_1 1 100 15 5  Intermediate m: array([[False, False, False, False, False, False, False], [False, False, False, False, False, False, True], [False, False, False, False, True, False, False]])
Question: I get the following error while attempting to a register a user with the help of DRF, dj-rest-auth and django-allauth: AttributeError at /api/v1/dj-rest-auth/registration/ 'EmailAddressManager' object has no attribute 'is_verified'  Here is the templates part of settings.py file:  TEMPLATES = [ { 'BACKEND': 'django.template.backends.django.DjangoTemplates', 'DIRS': [BASE_DIR / "templates"], 'APP_DIRS': True, 'OPTIONS': { 'context_processors': [ 'django.template.context_processors.debug', 'django.template.context_processors.request', 'django.contrib.auth.context_processors.auth', 'django.contrib.messages.context_processors.messages', "django.template.context_processors.request", ], }, }, ] EMAIL_BACKEND = "django.core.mail.backends.console.EmailBackend" SITE_ID = 1  and project level urls.py file:  urlpatterns = [ path('admin/', admin.site.urls), path('', include('apps.pages.front.urls')), path('api/v1/', include("apps.contacts.api.urls")), path('api-auth/', include("rest_framework.urls")), path("api/v1/dj-rest-auth/", include("dj_rest_auth.urls")), path("api/v1/dj-rest-auth/registration/", include("dj_rest_auth.registration.urls")), ]  And just in case if order of installed apps matter, here is my installed_apps: INSTALLED_APPS = [ 'django.contrib.admin', 'django.contrib.auth', 'django.contrib.contenttypes', 'django.contrib.sessions', 'django.contrib.messages', 'whitenoise.runserver_nostatic', 'django.contrib.staticfiles', 'django.contrib.sites', "rest_framework", "corsheaders", "rest_framework.authtoken", "allauth", "allauth.account", "allauth.socialaccount", "dj_rest_auth", 'dj_rest_auth.registration', 'drf_spectacular', 'django_filters', 'apps.pages.apps.PagesConfig', 'apps.accounts.apps.AccountsConfig', 'apps.contacts.apps.ContactsConfig', ]  (i'm keeping my apps in a dedicated apps folder, and have created a custom user model) Leaving the email field empty will successfully register a new user, but it won't work if i add an email.

Answer: This issue occurs when there is a version mismatch between your django, django-allauth, or dj-rest-auth packages. In my case, I was using the following versions: django = “^4.2.4” django-allauth = “^0.54.0” dj-rest-auth = “^5.0.2”  To resolve the issue, I downgraded dj-rest-auth to version “^5.0.1”. I recommend upgrading all three packages to their latest versions and it should work as intended.
Question: I am having multiple problems with making my pythonscript dartsapp.py executable for my pi. At first i tried to use chmod +x dartsapp.py  and ./dartsapp.py  but i got the error: -bash: ./dartsapp.py: cannot execute: required file not found  I also added the dir to PATH but still no luck. Afterwards i tried to use pyinstaller but when i try to call pyinstaller --version  i get the same -bash: pyinstaller: command not found  even though my PATH looks like this: maxdu@LEDpi:~/0306 $ echo $PATH /home/maxdu/0306:/home/maxdu/0306/dartsapp.py:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/local/games:/usr/games  What am i missing / doing wrong here? Update: ls -l /usr/bin/python3  gives me lrwxrwxrwx 1 root root 10 Apr 9 2023 /usr/bin/python3 -> python3.11  and head -n 3 dartsapp.py | cat -e  gives me #!/usr/bin/python3^M$ ^M$ # Python-App fM-CM-<r das PunktezM-CM-$hlen beim Darts^M$  and type python3  gives me python3 is hashed (/usr/bin/python3) 

Answer: head -n 3 dartsapp.py | cat -e  outputs: #!/usr/bin/python3^M$ ^M$ # Python-App fM-CM-<r das PunktezM-CM-$hlen beim Darts^M$  and so those ^Ms at the end of each line tells us that dartsapp.py has DOS line endings, see linux-bash-shell-script-error-cannot-execute-required-file-not-found for more information on this specific error and Why does my tool output overwrite itself and how do I fix it? for more information on DOS line endings in general. Both of those questions contain various fixes for this problem, starting with running dos2unix dartsapp.py.
Question: df = pl.DataFrame({'list_column': [['a.xml', 'b.xml', 'c', 'd'], ['e.xml', 'f.xml', 'g', 'h']]}) def func(x): return [y for y in x if '.xml' in y] df.with_columns(pl.col('list_column').map_elements(func, return_dtype=pl.List(pl.String)))  Is there a way to achieve the same without using map_elements?

Answer: There is an accepted request for list.filter()  https://github.com/pola-rs/polars/issues/9189  You can emulate the behaviour using list.eval() df.with_columns( pl.col('list_column').list.eval( pl.element().filter(pl.element().str.ends_with('.xml')) ) )  shape: (2, 1) ┌────────────────────┐ │ list_column │ │ --- │ │ list[str] │ ╞════════════════════╡ │ ["a.xml", "b.xml"] │ │ ["e.xml", "f.xml"] │ └────────────────────┘