eshangj/stackoverflow_q_and_a_sample huggingface dataset:
Question: I am currently trying to parse some data from bash tables, and I found strange behavior in parsing data if some columns is empty for example i have data like this containerName ipAddress memoryMB name numberOfCpus status --------------- --------------- ---------- ------- -------------- ---------- TEST_VM 192.168.150.111 8192 TEST_VM 4 POWERED_ON  and sometimes like this containerName ipAddress memoryMB name numberOfCpus status --------------- ----------- ---------- ---------------------- -------------- ----------- TEST_VM_second 3072 TEST_VM_second_renamed 1 POWERED_OFF  I tried with python and with bash, but same results, I need data "name" but when I am using bash for example awk '{print $4}' in first table it prints expected result: name ------- TEST_VM  but in second table in prints: name ---------------------- 1  same results with python: df_info = pd.read_table(StringIO(table), delim_whitespace=True) df_info = df_info.drop(0) pd.set_option('display.max_colwidth', None) print(df_info['name'], df_info['containerName'])  Output:  1 TEST_VM Name: name, dtype: object 1 TEST_VM Name: containerName, dtype: object 1 1 Name: name, dtype: object 1 TEST_VM_second Name: containerName, dtype: object  Maybe someone knows how to play around if ipaddress is empty field ?

Answer: Don't parse the file manually, take advantage of pandas.read_fwf: df_info = pd.read_fwf(StringIO(table), skiprows=[1], delim_whitespace=True) df_info.columns = df_info.columns.str.strip()  Output:  containerName ipAddre memoryMB name numberOfCpu tatu 0 TEST_VM_second 3072 TEST_VM_second_renamed 1 POWERED_OFF  Or, to get NaNs: df_info = pd.read_fwf(StringIO(table), skiprows=[1])  Output:  containerName ipAddress memoryMB name numberOfCpus status 0 TEST_VM_second NaN 3072 TEST_VM_second_renamed 1 POWERED_OFF
Question: I'm trying to find the peaks of a noisy signal using scipy.signal.find_peaks and I realised that I don't fully understand the difference between the threshold and prominence arguments. I understand that prominence is equivalent to topographical prominence, i.e. the height of a peak relative to the surrounding terrain. However, I don't quite understand in which ways the threshold argument is different from this. From the above link both arguments seem equivalent to me. What is exactly the difference between threshold and prominence in this case?

Answer: Threshold is about vertical distance to the samples right before and after. Prominence is about vertical distance to the deepest valley. Here's a visual explanation of the difference:  In this graph of cos(x), the peak has a threshold of 0.191, and a prominence of 2. What implications does this have?  Threshold is good at distinguishing peaks which go up for a single sample, then go down. Prominence is good at distinguishing peaks which go up and down over the course of several samples. A single threshold value gets more selective as the sampling rate goes up. However, prominence is independent of sampling rate. (As an example, try changing the number of samples from 21 to 41 in the example below, and see how threshold changes.) The parameter wlen controls how many samples are searched for the valley to determine prominence. If you set wlen=3, then prominence and threshold are the same. By default, it will search until the next larger peak. See documentation.  Code used to produce graphic import matplotlib.pyplot as plt import numpy as np import scipy.signal x = np.linspace(0, 4*np.pi, 21) y = np.cos(x) plt.plot(x, y) # Show threshold and prominence for peak print(scipy.signal.find_peaks(y, threshold=0, prominence=0))
Question: I want to be able to manage my Supabase project as a code, with regards to the SQL tables, the functions, auth, et cetera. The purpose is to allow myself to version control the project. I have gone through the official documentation and have not found anything pertaining to this. The closest approach I can find is simply version controlling all the SQL code being used - but that is not reassuring for production purposes.

Answer: Everything (except for some auth related features) you mentioned can be done with the Supabase CLI and it's migration capabilities. There is a guide on thw website explaining how to manage environments for this sort of setup https://supabase.com/docs/guides/cli/managing-environments.
Question: I am trying to understand how OR-TOOLS works and wanted to solve the assignment problem but with number of tasks (2 out of 4) as constraints. But my code does not work as expectected. I have tried: from ortools.sat.python import cp_model costs = [ [90, 80, 75, 70], [35, 85, 55, 65], [125, 95, 90, 95], [45, 110, 95, 115], [50, 100, 90, 100], ] num_workers = len(costs) num_tasks = len(costs[0]) # Model model = cp_model.CpModel() # Variables x = [] for i in range(num_workers): t = [] for j in range(num_tasks): t.append(model.NewBoolVar(f'x[{i},{j}]')) x.append(t) # Constraints # Each worker is assigned to exactly one task. for worker in range(num_workers): model.AddExactlyOne(x[worker][task] for task in range(num_tasks)) # Choosing number of tasks. for worker in range(num_workers): model.Add(sum([x[worker][task] for task in range(num_tasks)]) == 2) # Objective objective_terms = [] for i in range(num_workers): for j in range(num_tasks): objective_terms.append(costs[i][j] * x[i][j]) model.Minimize(sum(objective_terms)) # Solve solver = cp_model.CpSolver() status = solver.Solve(model) # Print solution. if status == cp_model.OPTIMAL or status == cp_model.FEASIBLE: print(f'Total cost = {solver.ObjectiveValue()}') print() for i in range(num_workers): for j in range(num_tasks): if solver.BooleanValue(x[i][j]): print( f'Worker {i} assigned to task {j} Cost = {costs[i][j]}') else: print('No solution found.')  So how do I define a number of tasks to use as a constraint? I get "No solution found" as output. But I expect that task 0 and task 3 will be selected and output should look like this Worker 0 assigned to task 1 Cost = 70 Worker 1 assigned to task 0 Cost = 35 Worker 2 assigned to task 1 Cost = 95 Worker 3 assigned to task 0 Cost = 45 Worker 4 assigned to task 0 Cost = 50  If we assign every worker to only one task and choose two tasks, this should be the optimal solution.

Answer: This can be achieved using channeling constraint Expressing @AirSquid's idea in CP-SAT syntax: task_assigned = [model.NewBoolVar(f"task_assigned{i}") for i in range(num_tasks)] for task_id in range(num_tasks): sum_expr = sum([x[worker][task_id] for worker in range(num_workers)]) model.Add(sum_expr >= 1).OnlyEnforceIf(task_assigned[task_id]) model.Add(sum_expr == 0).OnlyEnforceIf(task_assigned[task_id].Not()) model.Add(sum(task_assigned) == 2)  This gives the output: Total cost = 295.0 Worker 0 assigned to task 2 Cost = 75 Worker 1 assigned to task 0 Cost = 35 Worker 2 assigned to task 2 Cost = 90 Worker 3 assigned to task 0 Cost = 45 Worker 4 assigned to task 0 Cost = 50
Question: Here is an original dataframe,  country2 year 2017 2018 2019 country1 1 2 1 2 3 6 data_provider indicator prov_1 ind_a 45 30 22 30 30 30 prov_2 ind_a 30 30 30 30 25 30 ind_b 30 32 30 30 30 30 prov_3 ind_b 30 30 30 35 30 28  and I wish to filter the column and finally get a new dataframe, # item country2 # year 2017 2018 2019 # country1 1 2 1 2 3 6 # data_provider indicator # prov_1 ind_a 45.0 NaN 22.0 NaN NaN NaN # prov_2 ind_a NaN NaN NaN NaN NaN NaN # ind_b NaN 32.0 NaN NaN NaN NaN # prov_3 ind_b NaN NaN NaN NaN NaN NaN  you can get the original dataframe by, df = pd.DataFrame( data={"data_provider": ["prov_1", "prov_1", "prov_2", "prov_2", "prov_3", "prov_3"], "indicator": ["ind_a", "ind_a", "ind_a", "ind_b", "ind_b", "ind_b"], "unit": ["EUR", "EUR", "EUR", "EUR", "EUR", "EUR"], "year": ["2017", "2018","2019", "2017","2018","2019"], "country1": [1, 1, 3, 2, 2, 6], "country2": [45, 22, 25, 32, 35, 28] } ) df = df.pivot_table( index=['data_provider', 'indicator'], columns=['year', 'country1'], fill_value=30 ) df.columns.names = ['item', 'year', 'country1']  here is how I get the new dataframe,  locate the 2 group of target column labels x1 = df.columns[df.columns.get_level_values(level='year')=='2017'] x2 = df.columns[df.columns.get_level_values(level='year')=='2018']  get newdf1 with condition1 df[x1]>30 newdf1 = df[df[x1] > 30]  get newdf2 with condition2 df[x2]<30 newdf2 = df[df[x2] < 30]  update newdf2 with newdf1   newdf = newdf2.combine_first(newdf1)  in my solution, I first get 2 dataframe after filtering the original dataframe with different conditions and then combine them together. I am wondering if there's a one direct way to achieve the goal.

Answer: import pandas as pd df = pd.DataFrame( data={ "data_provider": ["prov_1", "prov_1", "prov_2", "prov_2", "prov_3", "prov_3"], "indicator": ["ind_a", "ind_a", "ind_a", "ind_b", "ind_b", "ind_b"], "unit": ["EUR", "EUR", "EUR", "EUR", "EUR", "EUR"], "year": ["2017", "2018","2019", "2017","2018","2019"], "country1": [1, 1, 3, 2, 2, 6], "country2": [45, 22, 25, 32, 35, 28] } ) df = df.pivot_table( index=['data_provider', 'indicator'], columns=['year', 'country1'], fill_value=30 ) df.columns.names = ['item', 'year', 'country1'] cond1 = (df.columns.get_level_values(level='year') == '2017') & df.gt(30) cond2 = (df.columns.get_level_values(level='year') == '2018') & df.lt(30) df_new = df.where(cond1 | cond2) print(df_new)  Output :  country2 year 2017 2018 2019 country1 1 2 1 2 3 6 data_provider indicator prov_1 ind_a 45.0 NaN 22.0 NaN NaN NaN prov_2 ind_a NaN NaN NaN NaN NaN NaN ind_b NaN NaN NaN NaN NaN NaN prov_3 ind_b NaN NaN NaN NaN NaN NaN
Question: so i am currently making the error handles for my discord bot, and when it returns that it has not all arguments, this is my code: @bot.event async def on_command_error(ctx, error): if isinstance(error, commands.MissingRequiredArgument): await ctx.send(f'❓ it seems like your entry `{ctx.message.content}` **does not contain all needed arguments!** try `+help {ctx.message.content.split()[0]}` for more info!')  as you can see, i want the user to be able to see a +help [COMMAND EXECUTED WITHOUT PREFIX] in the response, how would i do that? everything works, it‘s just the prefix in the ctx.message.content.split()[0] that i don‘t want to be there.. thanks in advance! (and btw, the response message is attached as picture!)

Answer: You can simply use the ctx.command.name attribute: @bot.event async def on_command_error(ctx, error): if isinstance(error, commands.MissingRequiredArgument): command = ctx.command.name await ctx.send(f'❓ it seems like your entry `{ctx.message.content}` **does not contain all needed arguments!** try `+help {command}` for more info!')
Question: I believe there are two ways of checking if a torch.Tensor has values all greater than 0. Either with .all() or all(), a Minimal Reproducible Example will illustrate my idea: import torch walls = torch.tensor([-1, 0, 1, 2]) result1 = (walls >= 0.0).all() # DIFFERENCE WITH BELOW??? result2 = all(walls >= 0.0) # DIFFERENCE WITH ABOVE??? print(result1) # Output: False print(result2) # Output: False  all() is builtin so I think I would prefer using that one, but most code I see on the internet uses .all() so I'm afraid there is unexpected behaviour. Are they both behaving the exact same?

Answer: all is Python builtin, meaning that it only works using extremely generic interfaces. In this case, all treats the tensor as an opaque iterable. It proceeds by iterating the elements of the tensor one-by-one, constructing a Python object for each one and then checking the truthiness of that Python object. That is slow, with several added layers of unnecessary inefficiency. In contrast, Tensor.all knows what a Tensor object is, and can operate on it directly. It only needs to directly scan the tensors internal storage. No iterator protocol function calls, no intermediate Python objects. Tensor.all will always be far more efficient, both in time and memory, that the builtin all.
Question: I am migrating my old code from pandas to polars. But i am not able to have a workaround for apply function. I have a function where i do some calculation from the data received from apply function where i have to use some column. My code In pandas for example: import pandas as pd data = {'A': [1, 2, 3], 'B': [4, 5, 6]} df = pd.DataFrame(data) self.var1 = 'A' self.var2 = 'B' def some_calculation(row): var = self.var1 var2 = self.var2 cal1 = row[var] + row[var2] cal2 = row[var] * row[var2] return list(zip(cal1 , cal2)) df['calc_data'] = df.apply(some_calculation, axis=1)  Now, after going through the polars documentation and using the apply function i found out that it doesn't pass the column name and due to this I am unable to do the calculations as my dataframe may have varying columns and i was not able to find a solution to this. Please help.

Answer: The solution is to stop thinking in terms of apply. You can do something like ( pl.from_pandas(df) .with_columns( calc_data= pl.concat_list( pl.col(var1)+pl.col(var2), pl.col(var1)*pl.col(var2) ) ) ) shape: (3, 3) ┌─────┬─────┬───────────┐ │ A ┆ B ┆ calc_data │ │ --- ┆ --- ┆ --- │ │ i64 ┆ i64 ┆ list[i64] │ ╞═════╪═════╪═══════════╡ │ 1 ┆ 4 ┆ [5, 4] │ │ 2 ┆ 5 ┆ [7, 10] │ │ 3 ┆ 6 ┆ [9, 18] │ └─────┴─────┴───────────┘  although you probably don't actually want the return to be a list so you'd do something more like ( pl.from_pandas(df) .with_columns( calc1=pl.col(var1)+pl.col(var2), calc2=pl.col(var1)*pl.col(var2) ) ) shape: (3, 4) ┌─────┬─────┬───────┬───────┐ │ A ┆ B ┆ calc1 ┆ calc2 │ │ --- ┆ --- ┆ --- ┆ --- │ │ i64 ┆ i64 ┆ i64 ┆ i64 │ ╞═════╪═════╪═══════╪═══════╡ │ 1 ┆ 4 ┆ 5 ┆ 4 │ │ 2 ┆ 5 ┆ 7 ┆ 10 │ │ 3 ┆ 6 ┆ 9 ┆ 18 │ └─────┴─────┴───────┴───────┘  If you are adamant about maintaining the anti-pattern that is rowwise iteration with names then you can use iter_rows(named=True) to get a generator of dicts for each row. Your function, as defined, doesn't work for me so I amended it to this... var1='A' var2='B' def some_calculation(row): cal1 = row[var1] + row[var2] cal2 = row[var1] * row[var2] return [cal1 , cal2]  in which case I get... df['calc_data'] = df.apply(some_calculation, axis=1) df A B calc_data 0 1 4 [5, 4] 1 2 5 [7, 10] 2 3 6 [9, 18]  Using the generator you can do: pldf=pl.from_pandas(df) pldf.with_columns( calc_data=pl.Series(some_calculation(x) for x in pldf.iter_rows(named=True)) ) shape: (3, 3) ┌─────┬─────┬───────────┐ │ A ┆ B ┆ calc_data │ │ --- ┆ --- ┆ --- │ │ i64 ┆ i64 ┆ list[i64] │ ╞═════╪═════╪═══════════╡ │ 1 ┆ 4 ┆ [5, 4] │ │ 2 ┆ 5 ┆ [7, 10] │ │ 3 ┆ 6 ┆ [9, 18] │ └─────┴─────┴───────────┘  That being said, there's little to be gained by "migrating" to polars if you are going to maintain rowwise iteration.
Question: I'd like to combine two iterators and yield the value(s) of the iterator that has the highest timestamp. Minimal example and expectations: # Outputs of these generators are "timestamps" def gen_even(): for x in range(0, 11, 2): yield x def gen_odd(): for x in sorted(list(range(1, 15, 2)) + [6]): yield x  Combining these two should result in in the following sequence [0, 1, 2, 3, 4, 5, (6, 6), 7, 8, 9, 10, 11, 13] I tried the following which runs into StopIteration after gen1 has been consumed. gen1 = gen_even() gen2 = gen_odd() def gen_both(gen1, gen2): first = next(gen1) second = next(gen2) while True: if first < second: yield first first = next(gen1) elif first == second: yield first, second first = next(gen1) second = next(gen2) else: yield second second = next(gen2) gen = gen_both(gen1, gen2) for i in gen: print(i)  Output: 0 1 2 3 4 5 (6, 6) 7 8 9 10 --------------------------------------------------------------------------- StopIteration Traceback (most recent call last) Cell In[8], line 11, in gen_both(gen1, gen2) 10 yield first ---> 11 first = next(gen1) 12 elif first == second: StopIteration: The above exception was the direct cause of the following exception: RuntimeError Traceback (most recent call last) Cell In[8], line 21 18 second = next(gen2) 20 gen = gen_both(gen1, gen2) ---> 21 for i in gen: 22 print(i) RuntimeError: generator raised StopIteration  How can I do this in Python?

Answer: You could avoid the stop iterator by handling it as follows: def gen_both(gen1, gen2): first = next(gen1, None) second = next(gen2, None) while True: if first is None and second is None: break elif first is None: yield second second = next(gen2, None) elif second is None: yield first first = next(gen1, None) else: if first < second: yield first first = next(gen1, None) elif first == second: yield first, second first = next(gen1, None) second = next(gen2, None) else: yield second second = next(gen2, None)
Question: I am currently trying to filter a pandas df with certain boolean columns for true, false and both types within a loop. For better explanation let's say we have the following df: df = pd.DataFrame({ 'a': [1, 2, 3, 5, 6, 7, 8, 9, 10], 'b': [True, True, True, False, False, True, False, True, False], 'c': [True, False, False, False, True, True, True, True, False], 'd': [35, 59, 12, 2, 19, 24, 33, 5, 11] })  What I want to do now is loop over the both boolean columns like so: loops = [[True, True], [True, any], [any, True], [any, any]] for loop in loops: df[(df['b'] == loop[0]) & (df['c'] == loop[1])]['d'].sum()  Unfortunately using any is not working, as well as bool. As anyone an idea how that could be archived? Thank you!

Answer: Using any doesn't make sense here, this is a function not a generic term that pandas would understand. Why not using a set for your conditions? T = {True} F = {False} ANY = {True, False} NONE = {} loops = [[T, T], [T, ANY], [ANY, T], [ANY, ANY]] for loop in loops: x = df[df['b'].isin(loop[0]) & df['c'].isin(loop[1])]['d'].sum() print(loop, x)  Output: [{True}, {True}] 64 [{True}, {False, True}] 135 [{False, True}, {True}] 116 [{False, True}, {False, True}] 200  Alternative using functions If you create custom functions, make sure they are vectorized. def istrue(x): return x == True # or just "return x" if you only have booleans def isfalse(x): return x == False # not using "~ x" to make the function generic def isany(x): return x | True def isnone(x): return x & False loops = [[istrue, istrue], [istrue, isany], [isany, istrue], [isany, isany]] for loop in loops: x = df[loop[0](df['b']) & loop[1](df['c'])]['d'].sum() print([x.__name__ for x in loop], x)  Output: ['istrue', 'istrue'] 64 ['istrue', 'isany'] 135 ['isany', 'istrue'] 116 ['isany', 'isany'] 200
Question: I have a requirements.txt that uses git+https URLs like git+https://github.com/myorg/myrepo@v1.0.0#egg=mypackage  This installs fine with pip install -r requirements.txt . But if I reference it in my setup.cfg: [options] install_requires = file: requirements.txt  and pip install -e . my package as an editable install from a local dir for development, pip fails with:  Getting requirements to build editable ... error error: subprocess-exited-with-error × Getting requirements to build editable did not run successfully. │ exit code: 1 [...snip...] setuptools.extern.packaging._tokenizer.ParserSyntaxError: Expected end or semicolon (after name and no valid version specifier) git+https://github.com/myorg/myrepo@v1.0.0#egg=mypackage  despite the same requirements.txt being valid for -r . What's up?

Answer: It looks like setuptools requires the package name to be supplied as a prefix, then the fetch URL after an @, e.g. # requirements.txt mypackage @ git+https://github.com/myorg/myrepo@v1.0.0#egg=mypackage  This form is accepted by pip install -e . with a setup.cfg that indirectly references the requirements.txt, and by pip install -r requirements.txt. It might be relevant that my project uses setuptools-scm.
Question: I'm trying to solve reading a zipfile from stdin in python, but I keep getting issues. What I want is to be able to run cat test.xlsx | python3 test.py and create a valid zipfile.ZipFile object without first writing a temporary file if possible. My initial approach was this, but ZipFile complained the file is not seekable,  import sys import zipfile zipfile.ZipFile(sys.stdin)  so I changed it around, but now it complains that this is not a valid zip file: import io import sys import zipfile zipfile.ZipFile(io.StringIO(sys.stdin.read()))  Can this be solved without writing the zip to a temporary file?

Answer: Zip files are binary data, not UTF-8 encoded text. You won't be able to read the file into a str with sys.stdin.read() without immediately hitting a UnicodeDecodeError: 'utf-8' codec can't decode byte ... error. Instead, you can access the underlying binary buffer object to read stdin as raw bytes. Pair that with BytesIO to get an in-memory seekable file-like object: zipfile.ZipFile(io.BytesIO(sys.stdin.buffer.read()))  Alternatively, if you provide a seekable stdin (for example, by redirecting stdin instead of streaming from a pipe), you can operate on sys.stdin.buffer directly: zipfile.ZipFile(sys.stdin.buffer)  paired with something like python3 test.py <test.xlsx  If you care to, you can select between the two depending on whether stdin is seekable by querying the IO object's seekable method: if sys.stdin.buffer.seekable(): zip_file = zipfile.ZipFile(sys.stdin.buffer) else: buffer = io.BytesIO(sys.stdin.buffer.read()) zip_file = zipfile.ZipFile(buffer) print(zip_file.filelist)
Question: I have a 2 dimensional array (10x10) that represents x & y positions. I'm writing a script that goes to each position, does some stuff, then moves to the next position. The most efficient way would be to start in one corner, scan at a constant x value (moving through y values), then at the end of all of the y values, moves over one x value and moves through y values in reverse order. Kind of like a snake pattern if that makes sense. My loop currently looks something like this: for x in x_values: for y in y_values: do_something() move to y + 1 move to x + 1 move back to first y value  As you can see, I'm completing a column (in y), advancing one x value, then rewinding back to the first y value. I'd like to run the y values in reverse for even number x values. What is the simplest way to do this?

Answer: This should solve your problem for x in x_values: for y in y_values: do_something() y_values.reverse()  After each iteration over y_values ​​, y_values ​​is reversed.
Question: I have a Dataframe and I am looking for a way to find the maximum number on uniquly matched pairs on two columns. If for example I limit my Dataframe to only these two columns:     X Y     1 a   1 b   2 c   2 d   3 b   3 a   4 c   4 d   4 e   5 e   5 c     The result shoud be:     X Y     1 a   2 c   3 b   4 d   5 e     So if I match 1 & a, then I cannot use 1 for column X anymore, nor a for Y anymore. The difficult part is to match the maximum number of pairs possible with this rule. Many thanks.

Answer: I would use a linear_sum_assignment on the crosstab: from scipy.optimize import linear_sum_assignment tmp = pd.crosstab(df['X'], df['Y']) idx, col = linear_sum_assignment(tmp, maximize=True) out = pd.DataFrame({'X': tmp.index[idx], 'Y': tmp.columns[col]})  Output:  X Y 0 1 a 1 2 c 2 3 b 3 4 d 4 5 e  Intermediate crosstab (tmp): Y a b c d e X 1 1 1 0 0 0 2 0 0 1 1 0 3 1 1 0 0 0 4 0 0 1 1 1 5 0 0 1 0 1  other example # df: here we removed the 3/b and 5/e pairs X Y 0 1 a 1 1 b 2 2 c 3 2 d 4 3 a 5 4 c 6 4 d 7 4 e 8 5 c # out X Y 0 1 b 1 2 d 2 3 a 3 4 e 4 5 c