eshangj/stackoverflow_q_and_a_sample huggingface dataset:
Question: How can I create the type column recycling a two-elements list ["lat","lon"]?    adresse coord type     "place 1" 48.943837 lat   "place 1" 2.387917 lon   "place 2" 37.843837 lat   "place 2" 6.387917 lon    As it would be automatically done in R with d$type <- c("lat","lon") Reprex : d0 = pl.DataFrame( { "adresse": ["place 1", "place 2"], "coord": [[48.943837, 2.387917], [37.843837, 6.387917]], } ) d1 = d0.explode("coord")  What I tried: d1 = d1.with_columns(type=pl.Series(["1","2"])) # ShapeError: unable to add a column of length 2 to a DataFrame of height 4 d1 = d1.join(pl.DataFrame({"id":["1", "2"]}), how="cross") # logically, 8 rows instead of 4 

Answer:  pl.int_range() and pl.len() to create a "row number". pl.Expr.over() to do it within adresse column.  ( d0.explode("coord") .with_columns( type = pl.int_range(pl.len()).over("adresse") ) )  shape: (4, 3) ┌─────────┬───────────┬──────┐ │ adresse ┆ coord ┆ type │ │ --- ┆ --- ┆ --- │ │ str ┆ f64 ┆ i64 │ ╞═════════╪═══════════╪══════╡ │ place 1 ┆ 48.943837 ┆ 0 │ │ place 1 ┆ 2.387917 ┆ 1 │ │ place 2 ┆ 37.843837 ┆ 0 │ │ place 2 ┆ 6.387917 ┆ 1 │ └─────────┴───────────┴──────┘  Or if you need polars.datatypes.Enum(): dtype = pl.Enum(["lat", "lon"]) ( d0.explode("coord") .with_columns( type = pl.int_range(pl.len()).over("adresse").cast(dtype) ) )  shape: (4, 3) ┌─────────┬───────────┬──────┐ │ adresse ┆ coord ┆ type │ │ --- ┆ --- ┆ --- │ │ str ┆ f64 ┆ enum │ ╞═════════╪═══════════╪══════╡ │ place 1 ┆ 48.943837 ┆ lat │ │ place 1 ┆ 2.387917 ┆ lon │ │ place 2 ┆ 37.843837 ┆ lat │ │ place 2 ┆ 6.387917 ┆ lon │ └─────────┴───────────┴──────┘  Alternatively, you can first create additional lists with pl.int_ranges() and then explode both lists together, so you don't need pl.Expr.over() aka window function. ( d0 .with_columns(type = pl.int_ranges(2)) # or using pl.col.coord.list.len() # .with_columns(type = pl.int_ranges(pl.col.coord.list.len())) .explode("coord", "type") .with_columns(type = pl.col.type.cast(dtype)) )
Question: I have a function that I'd like to take numpy arrays or floats as input. I want to keep doing an operation until some measure of error is less than a threshold. A simple example would be the following to divide a number or array by 2 until it's below a threshold (if a float), or until it's maximum is below a threshold (if an array). def f(x): #float version while x>1e-5: x = x/2 return x def f(x): #np array version while max(x)>1e-5: x = x/2 return x  Unfortunately, max won't work if I've got something that is not iterable, and x>1e-5 won't work if x is an array. I can't find anything to do this, except perhaps vectorize, but that seems to not be as efficient as I would want. How can I get a single function to handle both cases?

Answer: What about checking the type of input inside the function and adapt it ? def f(x): # for float or np.array if type(x) is float: x = x * np.ones(1) while np.max(x)>1e-5: x = x/2 return x
Question: I'm learning Python (using MU) and trying to implement it in Excel (with Python in Excel), however I have found some difficulties which I can't understand: Cell values are (just an example): C1= 1 C2= 3 D1: i1=xl("C1") #gives value 1  D2: i2=xl("C2") #gives value 3  F1: i3=0 #i3 is defined, just to see what is happening  F2: i4=0 #same as above  F6: if i1<i2: i3=i1+i2 else: i3=i1*i2 #comparing values there is no output  F9: i3 #outputs the value and seems to work well  What I can't understand is why is in this case value of F6 cell none (Python editor says "No output"), even if I adjust the code to: if i1<i2: i1+i2 else: i1*i2  If i write i1+i2  in a random python activated cell, it gives off value as expected, but inside the if statement, it doesn't work. What seems to work is adding if statement into function: def foo(x1, x2): if x1<x2: return x1+x2 elif x1>x2: return x1*x2 else: return x1 foo(i1, i2)  in that case function returns to a cell and cell has a value as expected. If there is no way to achieve cell value directly from if statement, I assume using a function is a way to go.

Answer: TL;DR Python in Excel will only evaluate the last expression or assignment as output. Option 1: Add i3 to the code block after the if statement as the last expression to be evaluated. if i1 < i2: i3 = i1 + i2 else: i3 = i1 * i2 i3 # add  Option 2: Use a conditional expression. i3 = i1 + i2 if i1 < i2 else i1 * i2  Option 3: Declare a function and call it (as in OP's "EDIT" section).  The "documentation" on Python in Excel (hereafter: PiE) seems rather meagre. However, this feature being a collaboration with Anaconda, there are some useful blogs. In 5 Quick Tips for Using Python in Excel, one can read:  So what constitutes a valid output for a cell? The execution of each Python cell works in a read–eval–print loop (REPL) fashion, similar to running Python in Jupyter Notebook cells. The last expression in the cell that will be evaluated (e.g., a Python object or the return value of a called Python function) will represent the output of the cell.  The comparison with Jupyter Notebook is instructive, but rather imprecise. Cf. InteractiveShell.ast_node_interactivity, which shows you that it uses 'last_expr' as the default. PiE, however, seems to use the equivalent of 'last_expr_or_assign'. Cf. Jupyter Notebook:  With PiE: x = 1 y = 2   Otherwise, it is true that Jupyter Notebook (by default) also does not output an expression that is part of a control flow like the if statement. But it can be made to do so for an expression with 'all':  It would appear that PiE simply does not allow this. As a result, it seems you are left with 3 options: Option 1: Add the variable at the end of the code block, after the if statement: x = 1 if 1 == 1: x = 2 x   Option 2: Use a conditional expression ("ternary operator", cf. this SO post): x = 1 x = 2 if 1 == 1 else x   Option 3: The workaround already provided by the OP; declare a function and call it.
Question: its an aws lambda function  try: with urllib.request.urlopen(api_url) as response: data=json.loads(response.read().decode()) print(json.dumps(data, indent = 4 )) except Exception as e : print(f"error reading data : {e} ") return {"statuscode":"500","body":"Error fetching data"} games_messages = [game_format(game) for game in data]  the error that I get :  "errorMessage": "'NoneType' object is not iterable", "errorType": "TypeError", "requestId": "11a9a46e-3abe-4f6e-8c38-278c0697238a", "stackTrace": [ " File \"/var/task/lambda_function.py\", line 69, in lambda_handler\n games_messages = [game_format(game) for game in data]\n", " File \"/var/task/lambda_function.py\", line 15, in game_format\n quarter_scores = ', '.join([f\"Q{q['Number']}: {q.get('AwayScore', 'N/A')}-{q.get('HomeScore', 'N/A')}\" for q in quarters])\n" ] }  the data that I fetched: 

Answer: The issue is not originating from the code block you provided, rather it is originating from line 15: quarter_scores = ', '.join([f"Q{q['Number']}: {q.get('AwayScore', 'N/A')}-{q.get('HomeScore', 'N/A')}" for q in quarters])  Error message states quarters is None. Indeed, in the data you fetched, quarters is null, (json.loads function turns null into None)  Solution: before iterating, make sure quarters is not None.
Question: I have the following pd.DataFrame match_id player_id round points A B C D E 5890 3750 1 10 0 0 0 3 1 5890 3750 2 10 0 0 0 1 0 5890 3750 3 10 0 8 0 0 1 5890 2366 1 9 0 0 0 5 0 5890 2366 2 9 0 0 0 5 0 5890 2366 3 9 0 0 0 2 0  I want to subtract the values of A, B, C, D and E of the two players and create two new columns that represent the number of points of the two players. My desired output looks as follows: match_id round points_home points_away A B C D E 5890 1 10 9 0 0 0 -2 1 5890 2 10 9 0 0 0 -4 0 5890 3 10 9 0 8 0 -2 1  Please advice

Answer: Use GroupBy.first with GroupBy.last first, subtract necessary columns with add points columns in concat: g = df.groupby(['match_id','round']) df1 = g.first() df2 = g.last() cols = ['A','B','C','D','E'] out = pd.concat([df1['points'].rename('points_home'), df2['points'].rename('points_away'), df1[cols].sub(df2[cols])], axis=1).reset_index() print (out) match_id round points_home points_away A B C D E 0 5890 1 10 9 0 0 0 -2 1 1 5890 2 10 9 0 0 0 -4 0 2 5890 3 10 9 0 8 0 -2 1  Alternative with MultiIndex with GroupBy.agg: df3 = (df.groupby(['match_id','round']) .agg(['first','last']) .rename(columns={'first':'home', 'last':'away'})) cols = ['A','B','C','D','E'] out = pd.concat([df3['points'].add_prefix('points_'), df3.xs('home', axis=1, level=1)[cols] .sub(df3.xs('away', axis=1, level=1)[cols])], axis=1).reset_index() print (out) match_id round points_home points_away A B C D E 0 5890 1 10 9 0 0 0 -2 1 1 5890 2 10 9 0 0 0 -4 0 2 5890 3 10 9 0 8 0 -2 1
Question: I have a linear system shown in the block diagram below.  This system is described with the following set of linear equations: err = inp - fb out = a * err fb = f * out  I would like to use sympy to compute the output (out) of the function of the input (inp). Thus, I would like to eliminate the variables err and fb. I would like some help as I been unable to figure out how to express what I want. So far I have: from sympy import symbols, Eq, solve inp, err, out, fb, a, f = symbols("inp err out fb a f") eqns = [ Eq(err, inp - fb), Eq(out, a * err), Eq(fb, f * out), ] solution = solve(eqns, [out]) solution # []  That clearly does not work. I thought perhaps simplify() might help here, but I don't know how to apply the simplify function to a system of equations. The result I am hoping to achieve is:  a out = ------ * inp 1 + af  Can anyone point me in the right direction?

Answer: I think you should specify all "outgoing" parameters, e.g., [out, fb, err], rather than [out] only, since [inp, a, f] could be treated as "constants" in this system of equations. from sympy import simplify, symbols, Eq, solve inp, err, out, fb, a, f = symbols("inp, err, out, fb, a, f") eqns = ( Eq(inp - fb, err), Eq(a * err, out), Eq(f * out, fb) ) solution = solve(eqns, [out, fb, err])  and you will see {err: inp/(a*f + 1), fb: a*f*inp/(a*f + 1), out: a*inp/(a*f + 1)}
Question: I am getting an SSL verification failed error when trying to send emails with the Sendgrid web api. I'm not even sure what cert it is trying to verify here. I have done all of the user and domain verification on my Sendgrid account and I am using very straightforward sending process. Here the error urllib.error.URLError: <urlopen error [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: self-signed certificate in certificate chain (_ssl.c:1000)>  Heres the code mailClient = SendGridAPIClient(os.environ.get('sendGridKey')) message = Mail( from_email=os.environ['sendGridEmail'], to_emails="dev@cribbi.co", subject="User Message for %s" %newMessage['chatId'], html_content = "<strong>" + newMessage['message']['text'] + "</strong>" ) mailRes = mailClient.send(message) 

Answer: Here is a link to the answer. It is not sendgrid specific but Python version specific. I was able to use the link provided in this answer to find the script that I needed to run to update the SSL certs for python. urllib and "SSL: CERTIFICATE_VERIFY_FAILED" Error
Question: I have a large (90k x 90k) numpy ndarray and I need to zero out a block of it. I have a list of about 30k indices that indicate which rows and columns need to be zero. The indices aren't necessarily contiguous, so a[min:max, min:max] style slicing isn't possible. As a toy example, I can start with a 2D array of non-zero values, but I can't seem to write zeros the way I expect. import numpy as np a = np.ones((6, 8)) indices = [2, 3, 5]  # I thought this would work, but it does not. # It correctly writes to (2,2), (3,3), and (5,5), but not all # combinations of (2, 3), (2, 5), (3, 2), (3, 5), (5, 2), or (5, 3) a[indices, indices] = 0.0 print(a) [[1. 1. 1. 1. 1. 1. 1. 1.] [1. 1. 1. 1. 1. 1. 1. 1.] [1. 1. 0. 1. 1. 1. 1. 1.] [1. 1. 1. 0. 1. 1. 1. 1.] [1. 1. 1. 1. 1. 1. 1. 1.] [1. 1. 1. 1. 1. 0. 1. 1.]]  # I thought this would fix that problem, but it doesn't change the array. a[indices, :][:, indices] = 0.0 print(a) [[1. 1. 1. 1. 1. 1. 1. 1.] [1. 1. 1. 1. 1. 1. 1. 1.] [1. 1. 1. 1. 1. 1. 1. 1.] [1. 1. 1. 1. 1. 1. 1. 1.] [1. 1. 1. 1. 1. 1. 1. 1.] [1. 1. 1. 1. 1. 1. 1. 1.]]  In this toy example, I'm hoping for this result. [[1. 1. 1. 1. 1. 1. 1. 1.] [1. 1. 1. 1. 1. 1. 1. 1.] [1. 1. 0. 0. 1. 0. 1. 1.] [1. 1. 0. 0. 1. 0. 1. 1.] [1. 1. 1. 1. 1. 1. 1. 1.] [1. 1. 0. 0. 1. 0. 1. 1.]]  I could probably write a cumbersome loop or build some combinatorically huge list of indices to do this, but it seems intuitive that this must be supported in a cleaner way, I just can't find the syntax to make it happen. Any ideas?

Answer: Based on hpaulj's comment, I came up with this, which works perfectly on the toy example. a[np.ix_(indices, indices)] = 0.0 print(a) [[1. 1. 1. 1. 1. 1. 1. 1.] [1. 1. 1. 1. 1. 1. 1. 1.] [1. 1. 0. 0. 1. 0. 1. 1.] [1. 1. 0. 0. 1. 0. 1. 1.] [1. 1. 1. 1. 1. 1. 1. 1.] [1. 1. 0. 0. 1. 0. 1. 1.]]  It also worked beautifully on the real data. It was faster than I expected and didn't noticeably increase memory consumption. Exhausting memory has been a constant concern with these giant arrays.
Question: I am plotting multiple line and scatter graphs on a single figure and axis. My code sets one variable called total_steel_area and then goes through a set of values of another variable called phi_x__h. It then calculates x and y values from these variables and places them in a list. It then plots the values. The code then moves to the next value of total_steel_area and repeats. The output graph is shown below. The diagonal lines connect the last value of one set of x,y values to the first value of the next set. My question is how can I remove this connecting line? My code is below: phi_N_bh = [] phi_M_bh2 = [] fig, ax = plt.subplots(dpi=100, figsize=(8,4)) for total_steel_area in np.arange(0.01,0.06,0.01): for phi_x__h in np.arange(0.1,2,0.1): phi_N__bh, phi_M__bh2 = calculate_phi_N__bh_and_phi_M__bh2(phi_x__h, lamb, alpha_cc, eta, f_ck, E_s, varepsilon_cu2, phi_d__h, phi_d2__h, f_yk, total_steel_area/2, total_steel_area/2) phi_N_bh.append(phi_N__bh) phi_M_bh2.append(phi_M__bh2) ax.plot(phi_M_bh2, phi_N_bh, c='b') ax.scatter(phi_M_bh2, phi_N_bh, c='b', s=10) ax.set_title('Column Design Chart for Rectangular Column with Symmetrical Compression and Tension Reinforcement') ax.set_xlabel('M/bh²') ax.set_ylabel('N/bh') ax.text(1-0.1, 1-0.1, f'f_ck = {f_ck}, f_yk = {f_yk}', horizontalalignment='center', verticalalignment='center', transform=ax.transAxes) ax.text(1-0.1, 1-0.2, f'd/h = {phi_d__h}, d2/h = {phi_d2__h}', horizontalalignment='center', verticalalignment='center', transform=ax.transAxes) ax.set_ylim(0) ax.set_xlim(0) ax.spines['top'].set_visible(False) ax.spines['right'].set_visible(False) ax.legend()  

Answer: You should initialize/reset your lists at each step of the outer loop: for total_steel_area in np.arange(0.01,0.06,0.01): phi_N_bh = [] phi_M_bh2 = [] for phi_x__h in np.arange(0.1,2,0.1): phi_N__bh, phi_M__bh2 = calculate_phi_N__bh_and_phi_M__bh2(phi_x__h, lamb, alpha_cc, eta, f_ck, E_s, varepsilon_cu2, phi_d__h, phi_d2__h, f_yk, total_steel_area/2, total_steel_area/2) phi_N_bh.append(phi_N__bh) phi_M_bh2.append(phi_M__bh2) ax.plot(phi_M_bh2, phi_N_bh, c='b') ax.scatter(phi_M_bh2, phi_N_bh, c='b', s=10)  There might be a vectorial way to compute your values, but this is difficult to guess without a reproducible example of the functions. Output:
Question: I'm trying to apply the TV filter to 2D array which includes many nan values: from skimage.restoration import denoise_tv_chambolle import numpy as np data_random = np.random.random ([100,100])*100 plt.imshow(data_random) plt.imshow(denoise_tv_chambolle(data_random)) data_random[20:30, 50:60] = np.nan data_random[30:40, 55:60] = np.nan data_random[40:50, 65:75] = np.nan plt.imshow(denoise_tv_chambolle(data_random))  The TV filter works well with all valid data, but will return a nan array if there're nan values. Original data:  Deonised data:  Data with nan values: 

Answer: You can use a masked array: m = np.isnan(data_random) data = np.ma.masked_array(np.where(m, 0, data_random), m) plt.imshow(denoise_tv_chambolle(data, weight=50))  Example output (with weight = 50):  For less artifacts you could fill the holes with the average instead of zero: m = np.isnan(data_random) data = np.ma.masked_array(np.where(m, np.nanmean(data_random), data_random), m) plt.imshow(denoise_tv_chambolle(data, weight=50))  Output:  Another option would be to fill the holes with the nearest neighbors (e.g. with distance_transform_edt), then denoise, then restore the NaNs: from scipy.ndimage import distance_transform_edt m = np.isnan(data_random) data_fill = data_random[tuple(distance_transform_edt(m, return_distances=False, return_indices=True))] plt.imshow(np.where(m, np.nan, denoise_tv_chambolle(data_fill, weight=50)))  Output:  Intermediate data_fill:
Question: I have a list of integers that is sorted and contains no duplicates: mylist = [2, 5,6,7, 11,12, 19,20,21,22, 37,38, 40]  I want a summarized text representation that shows groups of adjacent integers in a compressed form as a hyphenated pair. To be specific: Adjacent implies magnitude differing by 1. So an integer i is considered to be adjacent to j if j = i ± 1. Recall that the list is sorted. That means that adjacent integers will appear in monotonically increasing series in the list. So I want some elegant Python that will represent mylist as the string "2, 5-7, 11-12, 19-22, 37-38, 40,"  That is,  an isolated integer (example: 2, because the list contains neither 1 nor 3) is represented as 2, a group of adjacent integers (example: 19,20,21,22 because each member of the group differs from one other member by 1) is represented as ‹lowest›-‹highest›, that is 19-22,.  I can't believe this is a problem nobody has thought important enough to solve. Feel free to point me at a solution I have missed.

Answer: You can try this: mylist = [2, 5, 6, 7, 11, 12, 19, 20, 21, 22, 37, 38, 40] ans = [] ret = '' # assuming mylist is sorted for i in mylist: if len(ans) == 0 or ans[-1][1] < i - 1: #if the array is empty or we can't add current value to the last range ans.append([i, i]) # make a new range else: ans[-1][1] = i # add to last range for i in ans: # formating if i != ans[0]: ret = ret + ' ' if i[0] == i[1]: ret = f'{ret}{i[0]},' else: ret = f'{ret}{i[0]}-{i[1]},' print(ret)  Hope this helps!
Question: I am working with multiple Pandas DataFrames with a similar structure and would like to create reusable filters that I can define once and then apply or combine as needed. The only working solution I came up with so far feels clunky to me and makes it hard to combine filters with OR: import pandas as pd df = pd.DataFrame({"A":[1,1,2],"B":[1,2,3]}) def filter_A(df): return df.loc[df["A"]==1] def filter_B(df): return df.loc[df["B"]==2] print(filter_A(filter_B(df)).head())  I am hoping for something along the lines of filter_A = (df["A"]==1) filter_B = (df["B"]==2) print(df.loc[(filter_A) & (filter_B)])  but reusable after changing the df and also applicable to other DataFrames with the same columns. Is there any cleaner or more readable way to do this?

Answer: You can use the .eval() method, which allows for the evaluation of a string describing operations on dataframe columns:  Evaluate these string expressions on the dataframe df.  Combine the results of these evaluations using the bitwise AND operator (&), which performs element-wise logical AND operation.  Use the .loc accessor to filter the dataframe based on the combined condition.   filter_A = 'A == 1' filter_B = 'B == 2' df.loc[df.eval(filter_A) & df.eval(filter_B)]  Output:  A B 1 1 2
Question: I want to make a matplotlib figure that has two components:  A 2x2 "four pack" of subplots in the lower half of the figure  A subplot above the four pack that is the size of the four pack.   I have seen this answer where subplots can have different dimensions. How can that approach be tweaked when there are multiple columns and rows, however? Or is a completely different approach warranted? My final figure should be arranged something like this:  If there is a straightforward way to have a left-right oritentation, shown below, that would be helpful, too. 

Answer: You can use plt.subplot_mosaic or GridSpec. Someone else can write an answer using GridSpec, but here is how you'd do it using subplot_mosaic. For the large plot on top and 4 below it: import matplotlib.pyplot as plt fig, axs_dict = plt.subplot_mosaic("AA;AA;BC;DE")  If you want to put the large plot on the left and the four smaller on the right then just using this string: "AABC;AADE".
Question: I'm trying to add some new fields to an existing model and also a constraint related to those new fields: class User(models.Model): username = models.CharField(max_length=32) # New fields ################################## has_garden = models.BooleanField(default=False) garden_description = models.CharField( max_length=32, null=True, blank=True, ) class Meta: constraints = [ models.CheckConstraint( check=Q(has_garden=models.Value(True)) & Q(garden_description__isnull=True), name="garden_description_if_has_garden", ) ]  The problem is that when I run my migrations I get the following error: django.db.utils.IntegrityError: check constraint "garden_description_if_has_garden" is violated by some row  But I don't understand how the constraint is being violated if no User has a has_garden, the field is just being created and also its default value is False 🤔. I'm using django 3.2 with postgresql. What is the proper way to add this constraint? If it's of any use here's the autogenerated migration: # Generated by Django 3.2.25 on 2025-01-15 23:52 import django.db.models.expressions from django.db import migrations, models class Migration(migrations.Migration): dependencies = [ ("some_app", "0066_user"), ] operations = [ migrations.AddField( model_name="user", name="garden_description", field=models.CharField(blank=True, max_length=32, null=True), ), migrations.AddField( model_name="user", name="has_garden", field=models.BooleanField(default=False), ), migrations.AddConstraint( model_name="user", constraint=models.CheckConstraint( check=models.Q( ("has_garden", django.db.models.expressions.Value(True)), ("garden_description__isnull", True), ), name="garden_description_if_has_garden", ), ), ] 

Answer: Far as I can see your constraint is trying to make sure that all instances of user have has_garden=True, which would cause a violation if a user has has_garden=False. Here we add a constraint to either check if has_garden is true and garden_description is not null, or if has_garden is false and garden_descriptionn is null class User(models.Model): username = models.CharField(max_length=32) # New fields ################################## has_garden = models.BooleanField(default=False) garden_description = models.CharField( max_length=32, null=True, blank=True, ) class Meta: constraints = [ models.CheckConstraint( check=( Q(has_garden=models.Value(True)) & Q(garden_description__isnull=False) | ( Q(has_garden=models.Value(False)) & Q(garden_description__isnull=True) ) ), name="garden_description_if_has_garden", ) ]