eshangj/stackoverflow_q_and_a_sample huggingface dataset:
Question: Considering I have 2 dataframes as shown below (DF1 and DF2), I need to compare DF2 with DF1 such that I can identify all the Matching, Different, Missing values for all the columns in DF2 that match columns in DF1 (Col1, Col2 & Col3 in this case) for rows with same EID value (A, B, C & D). I do not wish to iterate on each row of a dataframe as it can be time-consuming. Note: There can around 70 - 100 columns. This is just a sample dataframe I am using. DF1  EID Col1 Col2 Col3 Col4 0 A a1 b1 c1 d1 1 B a2 b2 c2 d2 2 C None b3 c3 d3 3 D a4 b4 c4 d4 4 G a5 b5 c5 d5  DF2  EID Col1 Col2 Col3 0 A a1 b1 c1 1 B a2 b2 c9 2 C a3 b3 c3 3 D a4 b4 None  Expected output dataframe  EID Col1 Col2 Col3 New_Col 0 A a1 b1 c1 Match 1 B a2 b2 c2 Different 2 C None b3 c3 Missing in DF1 3 D a4 b4 c4 Missing in DF2 

Answer: Firstly, you will need to filter df1 based on df2. new_df = df1.loc[df1['EID'].isin(df2['EID']), df2.columns] EID Col1 Col2 Col3 0 A a1 b1 c1 1 B a2 b2 c2 2 C None b3 c3 3 D a4 b4 c4  Next, since you have a big dataframe to compare, you can change both the new_df and df2 to numpy arrays. array1 = new_df.to_numpy() array2 = df2.to_numpy()  Now you can compare it row-wise using np.where new_df['New Col'] = np.where((array1 == array2).all(axis=1),'Match', 'Different') EID Col1 Col2 Col3 New Col 0 A a1 b1 c1 Match 1 B a2 b2 c2 Different 2 C None b3 c3 Different 3 D a4 b4 c4 Different  Finally, to convert the row with None value, you can use df.loc and df.isnull new_df.loc[new_df.isnull().any(axis=1), ['New Col']] = 'Missing in DF1' new_df.loc[df2.isnull().any(axis=1), ['New Col']] = 'Missing in DF2' EID Col1 Col2 Col3 New Col 0 A a1 b1 c1 Match 1 B a2 b2 c2 Different 2 C None b3 c3 Missing in DF1 3 D a4 b4 c4 Missing in DF2
Question: So to use email as username I override the build-in User model like this (inspired by Django source code) models.py class User(AbstractUser): username = None email = models.EmailField(unique=True) objects = UserManager() USERNAME_FIELD = "email" REQUIRED_FIELDS = [] def __str__(self): return self.email  admin.py @admin.register(User) class UserAdmin(admin.ModelAdmin): fieldsets = ( (None, {"fields": ("email", "password")}), (("Personal info"), {"fields": ("first_name", "last_name")}), ( ("Permissions"), { "fields": ( "is_active", "is_staff", "is_superuser", "groups", "user_permissions", ), }, ), (("Important dates"), {"fields": ("last_login", "date_joined")}), ) add_fieldsets = ( ( None, { "classes": ("wide",), "fields": ("email", "password1", "password2"), }, ), ) list_display = ("email", "is_active", "is_staff", "is_superuser") list_filter = ("is_active", "is_staff", "is_superuser") search_fields = ("email",) ordering = ("email",) filter_horizontal = ("groups", "user_permissions",)  But this is how it looks like when I go to Admin site to change a user:  Password is visible and not hashed and no link to change password form. Comparing to what it looks like on a default Django project:  Password is not visible and there's a link to change password form So clearly I'm missing something but I can't figure out what it is.

Answer: You are not able to see the password in hashed state because the password field is a CharField which renders it as normal text field. In Django's admin side there's a field called ReadOnlyPasswordHashField in django.contrib.auth.forms which renders the password field to be in hashed state with password change link. Django's UserAdmin uses different form classes for user creation and updation.  form = UserChangeForm add_form = UserCreationForm change_password_form = AdminPasswordChangeForm  To edit user details UserAdmin uses form = UserChangeForm(source code) where the password field is set as ReadOnlyPasswordHashField(source code). class UserChangeForm(forms.ModelForm): password = ReadOnlyPasswordHashField( label=_("Password"), help_text=_( "Raw passwords are not stored, so there is no way to see this " "user’s password, but you can change the password using " '<a href="{}">this form</a>.' ), )  So, Just by inheriting from UserAdmin from django.contrib.auth.admin would make the password to be in hashed state with all the other essentials as seen in default admin site for users. OR you could simply import UserChangeForm from django.contrib.auth.forms and set form = UserChangeForm in custom UserAdmin from django.contrib.auth.forms import UserChangeForm,AdminPasswordChangeForm # code @admin.register(User) class UserAdmin(admin.ModelAdmin): # code form = UserChangeForm change_password_form = AdminPasswordChangeForm # code
Question: I see the parameters for running a default-settings pytest configuration are as follows: Launching pytest with arguments payments/tests/test_edd_countries.py --no-header --no-summary -q in payments/tests  I would like to remove all of those parameters specifically: --no-header --no-summary -q  How can that be achieved given the Runtime configuration in Edit Configurations does not even show them? 

Answer: The parameters --no-header --no-summary -q are added as an IDE setting (they aren't set in Run Configurations). They can be configured by going to File > Settings > Advanced Settings > Python and checking the option Pytest: do not add "--no-header --no-summary -q" as shown in the screenshot:
Question: With this line: print(sum(tuple), len(tuple), sum(tuple)/len(tuple))  Will python cache sum(tuple) in the 0 index and use it in the average calculation (2 index)? Or will with calculate sum(tuple) again?

Answer: Python won't perform this optimization for you. You can see this by defining your own function instead of sum and observing the side effect: import functools def my_sum(x): print('my sum') return functools.reduce(lambda a, b: a + b, x) tup = (1, 2, 3) print(my_sum(tup), len(tup), my_sum(tup)/len(tup))  If you run this snippet, you'll see the phrase "my sum" printed twice, proving the call to my_sum isn't optimized out. Having said that, you could implement this optimization yourself by using functools' cache: import functools @functools.cache def my_sum(x): print('my sum') return functools.reduce(lambda a, b: a + b, x) tup = (1, 2, 3) print(my_sum(tup), len(tup), my_sum(tup)/len(tup))  (or by using the := operator as Andrej Kesely suggests in the comments)
Question: I find it hard to explain with words what I want to achieve, so please don't judge me for showing a simple example instead. I have a table that looks like this:     main_col some_metadata value     this True 10   this False 3   that True 50   that False 10   other True 20   other False 5     I want to normalize this data separately for each case of main_col. For example, if we're to choose min-max normalization and scale it to range [0; 100], I want the output to look like this:     main_col some_metadata value (normalized)     this True 100   this False 30   that True 100   that False 20   other True 100   other False 25     Where for each case of main_col, the highest value is scaled to 100 and another value is scaled in respective proportion.

Answer: You can use groupby.transform('max') to get the max per group, then normalize in place: df['value'] /= df.groupby('main_col')['value'].transform('max').div(100)  or: df['value'] *= df.groupby('main_col')['value'].transform('max').rdiv(100)  output:  main_col some_metadata value 0 this True 100.0 1 this False 30.0 2 that True 100.0 3 that False 20.0 4 other True 100.0 5 other False 25.0
Question: when I run Django project or any code related to Postgres : Referenced from: '/Users/mahmoudnasser/.local/share/virtualenvs/wyspp_backend-PwdII1PB/lib/python3.8/site-packages/psycopg2/_psycopg.cpython-38-darwin.so' Reason: tried: '/opt/homebrew/opt/postgresql/lib/libpq.5.dylib' (no such file), '/usr/local/lib/libpq.5.dylib' (no such file), '/usr/lib/libpq.5.dylib' (no such file)  I tried many solutions online but none of them worked. Note: I use MacOS

Answer: To solve this problem just run the following command: sudo mkdir -p /usr/local/lib && sudo ln -s /opt/homebrew/opt/postgresql@14/lib/postgresql@14/libpq.5.dylib /usr/local/lib/libpq.5.dylib
Question: What would provide better performance using filtering conditions with Q in django ORM or simply fetching unfiltered objects and comparing in python. employee_qs = employee.objects.filter(state=States.ACTIVE, topic_assn__topic_id=instance.ss_topic_id).select_related('c_data').filter( Q(c_data__is_null=True) | Q(c_budget__gt=F('c_data__budget_spent') + offset_amt))  V/s  employee_qs = employee.objects.filter(state=States.ACTIVE, topic_assn__topic_id=instance.ss_topic_id).select_related('c_data') for employee in employee_qs: if not employee.c_data or float(employee.budget)-employee.c_data.budget_spent > offset_amt: #do something...  Which of these two choices would be better performance wise?

Answer: TLDR: Q objects will be faster. Why? Well filtering done with Q object will be done on the SQL server (either PostgreSQL, MariaDB). So two aspects should be considered:  with Q objects, unfiltered data will not be transferred from your database to your django server (less data over the network make things faster) ; Q objects are converted to SQL WHERE clause, depending on how your table is configured (for instance, the presence of indexes on the columns that you are filtering) the SQL server will use these indexes to filter data, which is much faster than on server side. Also, SQL servers are written in languages that are much faster than Python.
Question: Similar to this question, I would like to compute the time difference between rows of a dataframe. Unlike that question however, the difference should be by groupby id. So foe example, this dataframe: df = pd.DataFrame( {'id': [6,6,6,6,6,10,10,10,10,10], 'timestamp': ['2016-04-01 00:04:00','2016-04-01 00:04:20','2016-04-01 00:04:30', '2016-04-01 00:04:35','2016-04-01 00:04:54','2016-04-30 13:04:59', '2016-04-30 13:05:00','2016-04-30 13:05:12','2016-04-30 13:05:20', '2016-04-30 13:05:51']} ) df.head() id timestamp 0 6 2016-04-01 00:04:00 1 6 2016-04-01 00:04:20 2 6 2016-04-01 00:04:30 3 6 2016-04-01 00:04:35 4 6 2016-04-01 00:04:54 5 10 2016-04-30 13:04:59 6 10 2016-04-30 13:05:00 7 10 2016-04-30 13:05:12 8 10 2016-04-30 13:05:20 9 10 2016-04-30 13:05:51  Then I want to create a column ΔT for the differences, like so: df['timestamp'] = pd.to_datetime(df['timestamp'], format='%Y-%m-%d %H:%M:%S') df['ΔT'] = df.groupby('id').index.to_series().diff().astype('timedelta64[s]') AttributeError: 'DataFrameGroupBy' object has no attribute 'index'  Intended output:  id timestamp ΔT 0 6 2016-04-01 00:04:00 0 1 6 2016-04-01 00:04:20 20 2 6 2016-04-01 00:04:30 10 3 6 2016-04-01 00:04:35 5 4 6 2016-04-01 00:04:54 19 5 10 2016-04-30 13:04:59 0 6 10 2016-04-30 13:05:00 1 7 10 2016-04-30 13:05:12 12 8 10 2016-04-30 13:05:20 8 9 10 2016-04-30 13:05:51 31 

Answer: df.groupby('id')['timestamp'].diff().dt.total_seconds().fillna(0)
Question: I was making some experimentations with the OpenCV function cv2.warpPerspective when I decided to code it from scratch to better understand its pipeline. Though I followed (hopefully) every theoretical step, it seems I am still missing something and I am struggling a lot to understand what. Could you please help me? SRC image (left) and True DST Image (right) Output of the cv2.warpPerspective overlapped on the True DST # Invert the homography SRC->DST to DST->SRC hinv = np.linalg.inv(h) src = gray1 dst = np.zeros(gray2.shape) h, w = src.shape # Remap back and check the domain for ox in range(h): for oy in range(w): # Backproject from DST to SRC xw, yw, w = hinv.dot(np.array([ox, oy, 1]).T) # cv2.INTER_NEAREST x, y = int(xw/w), int(yw/w) # Check if it falls in the src domain c1 = x >= 0 and y < h c2 = y >= 0 and y < w if c1 and c2: dst[x, y] = src[ox, oy] cv2.imshow(dst + gray2//2)  Output of my code PS: The output images are the overlapping of Estimated DST and the True DST to better highlight differences.

Answer: Your issue amounts to a typo. You mixed up the naming of your coordinates. The homography assumes (x,y,1) order, which would correspond to (j,i,1). Just use (x, y, 1) in the calculation, and (xw, yw, w) in the result of that (then x,y = xw/w, yw/w). the w factor mirrors the math, when formulated properly. Avoid indexing into .shape. The indices don't "speak". Just do (height, width) = src.shape[:2] and use those. I'd recommend to fix the naming scheme, or define it up top in a comment. I'd recommend sticking with x,y instead of i,j,u,v, and then extend those with prefixes/suffixes for the space they're in ("src/dst/in/out"). Perhaps something like ox,oy for iterating, just xw,yw,w for the homography result, which turns into x,y via division, and ix,iy (integerized) for sampling in the input? Then you can use dst[oy, ox] = src[iy, ix]
Question: I am trying to write a regex pattern for phone numbers consisting of 9 fixed digits. I want to identify numbers that have two numbers alternating for four times such as 5XYXYXYXY I used the below sample number = 561616161  I tried the below pattern but it is not accurate ^5(\d)(?=\d\1).+  can someone point out what i am doing wrong?

Answer: I would use: ^(?=\d{9}$)\d*(\d)(\d)(?:\1\2){3}\d*$  Demo Here is an explanation of the pattern:  ^ from the start of the number (?=\d{9}$) assert exactly 9 digits \d* match optional leading digits (\d) capture a digit in \1 (\d) capture another digit in \2 (?:\1\2){3} match the XY combination 3 more times \d* more optional digits $ end of the number
Question: I have dataframe  id webpage 1 google 2 bing 3 google 4 google 5 yahoo 6 yahoo 7 google 8 google  Would like to count the groups like  id webpage count 1 google 1 2 bing 2 3 google 3 4 google 3 5 yahoo 4 6 yahoo 4 7 google 5 8 google 5  I have tried using the cumcount or ngroup when using groupby it is grouping all occurrence.

Answer: I believe you need to cumsum() over the state transitions. Every time webpage differs from the previous row you increase your count. df["count"] = (df.webpage != df.webpage.shift()).cumsum()
Question: I have a string for items separated by comma. Each item is surrounded by quotes ("), but the items can also contain commas (,). So using split(',') creates problems. How can I split this text properly in Python? An example of such string "coffee", "water, hot" What I want to achieve ["coffee", "water, hot"]

Answer: import ast s = '"coffee", "water, hot"' result = ast.literal_eval(f'[{s}]') print(result)
Question: Trying to freeze the first two rows and first column with openpyxl, however, whenever doing such Excel says that the file is corrupted and there is no freeze. Current code: workbook = openpyxl.load_workbook(path) worksheet = workbook[first_sheet] freeze_panes = Pane(xSplit=2000, ySplit=3000, state="frozen", activePane="bottomRight") worksheet.sheet_view.pane = freeze_panes  Took a look at the documentation, however, there is little explanation on parametere setting. Desired output:  Came across this answer, however, it fits a specific use case, hence, wanted to make a general question for future reference: How to split Excel screen with Openpyxl?

Answer: To freeze the first two rows and first column, use the sample code below... ws.freeze_panes works. Note that, like you would do in excel, select the cell above and left of which you want to freeze. So, in your case, the cell should be B3. Hope this is what you are looking for. import openpyxl wb=openpyxl.load_workbook('Sample.xlsx') ws=wb['Sheet1'] mycell = ws['B3'] ws.freeze_panes = mycell wb.save('Sample.xlsx')
Question: I want to drop all columns whose name starts by 'var' and whose content is 'None'. Sample of my dataframe: id var1 var2 newvar1 var3 var4 newvar2 1 x y dt None f None  Dataframe that I want: id var1 var2 newvar1 var4 newvar2 1 x y dt f None  I want to do this for several files and I do not know how many 'var' I have in all of them. My dataframe has only one row. Here is the code that I tried: for i in range(1,300): df.drop(df.loc[df['var'+str(i)] == 'None' ].index, inplace=True)  Error obtained: KeyError: 'var208'  I also tried: df.drop(df.loc[df['var'+str(i) for i in range(1,300)] == 'None'].index, inplace=True)  SyntaxError: invalid syntax  Could anyone help me improve my code?

Answer: Your error occurs because you have no column with that name. You can use df.columns to get a list of available columns, check if the name .startswith("var") and use df[col].isnull().all() to check if all values are None. import pandas as pd df = pd.DataFrame(columns=["id", "var1", "var2", "newvar1", "var3", "var4", "newvar2"], data=[[1, "x", "y", "dt", None, "f", None]]) df.drop([col for col in df.columns if col.startswith("var") and df[col].isnull().all()], axis=1, inplace=True)
Question: I am trying to install poetry using the following command curl -sSL https://install.python-poetry.org | python3 -  but it is failing with the following exception: Exception: This build of python cannot create venvs without using symlinks Below is the text detailing the error Retrieving Poetry metadata # Welcome to Poetry! This will download and install the latest version of Poetry, a dependency and package manager for Python. It will add the `poetry` command to Poetry's bin directory, located at: /Users/DaftaryG/.local/bin You can uninstall at any time by executing this script with the --uninstall option, and these changes will be reverted. Installing Poetry (1.2.1): Creating environment Traceback (most recent call last): File "<stdin>", line 940, in <module> File "<stdin>", line 919, in main File "<stdin>", line 550, in run File "<stdin>", line 571, in install File "/Library/Developer/CommandLineTools/Library/Frameworks/Python3.framework/Versions/3.9/lib/python3.9/contextlib.py", line 117, in __enter__ return next(self.gen) File "<stdin>", line 643, in make_env File "<stdin>", line 629, in make_env File "<stdin>", line 309, in make File "/Library/Developer/CommandLineTools/Library/Frameworks/Python3.framework/Versions/3.9/lib/python3.9/venv/__init__.py", line 66, in __init__ self.symlinks = should_use_symlinks(symlinks) File "/Library/Developer/CommandLineTools/Library/Frameworks/Python3.framework/Versions/3.9/lib/python3.9/venv/__init__.py", line 31, in should_use_symlinks raise Exception("This build of python cannot create venvs without using symlinks") Exception: This build of python cannot create venvs without using symlinks  I already have symlinks installed so not having that does not seem to be the problem. Would anyone know the cause of this error?

Answer: Not the best solution, but you can install it using Homebrew, if you have it. That's what I did. brew install poetry
Question: Sometimes as I'm using pdb, I would like to save the output to a file, pdbSaves.txt. For example I would want to do something like pp locals() >> pdbSaves.txt, which actually gives *** NameError: name 'pdbSaves' is not defined. What is the correct way to do this?

Answer: In Python 3 the ">>" symbol is no longer usable with the regular "print" for redirecting the output. I had not before used it from inside the PDB, but, certainly, the support for it was removed at the same time. What you have to do is to use the regular way of output to file with the new print function - or, if you want to do pretty print (pp does that), with the pprint.pprint function. (Pdb) from pprint import pprint as ppr (Pdb) file = open("x.txt", "wt") (Pdb) ppr("mystuff", stream=file)  Or, for regular printing, the parameter name for the output file is file rather than stream (the advantage is that the import statement is not needed): (Pdb) print ("mystuff", file=file)  Also, either these methods and the Python 2 >> way require the target to be an open file, not a string with the filename:
Question: Please find below my input/output : INPUT : dico = {'abc': 'val1=343, val2=935', 'def': 'val1=95, val2=935', 'ghi': 'val1=123, val2=508'}  OUTPUT (desired) :  I tried with pd.DataFrame.from_dict(dico, index=dico.keys) but unfortunately I got an error.  TypeError: DataFrame.from_dict() got an unexpected keyword argument 'index'  Do you have any suggestions please ?

Answer: Let's use a regex pattern to find the matching pairs corresponding to each value in the input dictionary then convert the pairs to dict and create a new dataframe import re pd.DataFrame([dict(re.findall(r'(\S+)=(\d+)', v)) for k, v in dico.items()], dico)  Alternative pandas only approach with extractall (might be slower): pd.Series(dico).str.extractall(r'(\S+)=(\d+)').droplevel(1).pivot(columns=0, values=1)  Result  val1 val2 abc 343 935 def 95 935 ghi 123 508
Question: we have a variable named location. location=[["world", 'Live'], ["alpha",'Live'], ['hello', 'Scheduled'],['alpha', 'Live'], ['just', 'Live'], ['alpha','Scheduled'], ['alpha', 'Live']]  i want to find the first index and count occurrence of list["alpha",'Live'] in location. i tried the following: index= [location.index(i) for i in location if i ==["alpha", 'Live'] ] count = [location.count(i) for i in location if i ==["alpha",'Live'] ] print('index',index) print('count', count)  this returns: index [1, 1, 1] count [3, 3, 3] but is there a way to find both first index, count simultaneously using list comprehension. expected output: index, count = 1, 3

Answer: does this solve you problem? location=[["world", 'Live'], ["alpha",'Live'], ['hello', 'Scheduled'],['alpha', 'Live'], ['just', 'Live'], ['alpha','Scheduled'], ['alpha', 'Live']] index= location.index(["alpha",'Live']) count = location.count(["alpha",'Live']) print('index',index) print('count', count)  if ['alpha','live'] is not found, find the first ['alpha',??] and print its index and count. location = [["world", 'Live'], ["alpha", 'Live'], ['hello', 'Scheduled'], [ 'alpha', 'Live'], ['just', 'Live'], ['alpha', 'Scheduled'], ['alpha', 'Live']] key = ["alpha", 'Lsive'] count = location.count(key) if count: index = location.index(key) print('count', count) print('index', index) else: for i in location: if i[0] == key[0]: key = i count = location.count(key) index = location.index(key) print('count', count) print('index', index) else: print('not found')  cleaner code by @yadavender yadav location = [["alpha", 'Scheduled'], ["alpha", 'Live'], ['hello', 'Scheduled'], [ 'alpha', 'Live'], ['just', 'Live'], ['alpha', 'Scheduled'], ['alpha', 'Live']] key = ["alpha", 'Scheduled'] count = location.count(key) if count: index = location.index(key) else: index=[location.index(i) for i in location if i[0]=="alpha"][0] print('count', count) print('index', index)
Question: How do I check if ANY key is pressed? this is how I know to detect one key: import keyboard # using module keyboard while True: # making a loop if keyboard.is_pressed('a'): # if key 'q' is pressed print('You Pressed A Key!') break # finishing the loop  How do I check if any key (not just letters) is pressed? For example, if someone presses the spacebar it works, the same for numbers and function keys, etc.

Answer: while True: # Wait for the next event. event = keyboard.read_event() if event.event_type == keyboard.KEY_DOWN: print(event.name) # to check key name  Press any key and get the key name.