eshangj/stackoverflow_q_and_a_sample huggingface dataset:
Question: I have lists where each entry is representing a nested structure, where / represents each level in the structure. ['a','a/b/a','a/b','a/b/d',....]  I want to take such a list and return an index list where each level is sorted in alphabetical order. If we had the following list ['a','a/b','a/b/a','a/c','a/c/a','b']  It represents the nested structure 'a': #1 'b': #1.1 'a': ... #1.1.1 'c': #1.2 'a': ... #1.2.1 'b' : ... #2  I am trying to get the output  ['1','1.1','1.1.1', '1.2','1.2.1','2']  But I am having real issue on how to tackle the problem, would it be solved recursively? Or what would be a way to solve this for any generic list where each level is separated by /? The list is originally not necessarily sorted, and each level can be any generic word.

Answer: Since the goal is to simply convert the paths to indices according to their respective positions against other paths of the same prefix, there is no need to build a tree at all. Instead, iterate over the paths in alphabetical order while using a dict of sets to keep track of the prefixes at each level of paths, and join the lengths of sets at each level for output: def indices(paths): output = {} names = {} for index, path in sorted(enumerate(paths), key=lambda t: t[1]): counts = [] prefixes = tuple(path.split('/')) for level, name in enumerate(prefixes): prefix = prefixes[:level] names.setdefault(prefix, set()).add(name) counts.append(len(names[prefix])) output[index] = '.'.join(map(str, counts)) return list(map(output.get, range(len(output))))  so that: print(indices(['a', 'a/b', 'a/b/a', 'a/c', 'a/c/a', 'b'])) print(indices(['a', 'c', 'b', 'a/b'])) print(indices(['a/b/c/d', 'a/b/d', 'a/b/c'])) print(indices(['abc/d', 'bcc/d'])) print(indices(['apple/cat','apple/dog', 'banana/dog']))  outputs: ['1', '1.1', '1.1.1', '1.2', '1.2.1', '2'] ['1', '3', '2', '1.1'] ['1.1.1.1', '1.1.2', '1.1.1'] ['1.1', '2.1'] ['1.1', '1.2', '2.1']  Demo: https://replit.com/@blhsing/StainedMassivePi
Question: I have this JSON/DICT in Python and I need to pass it to a datframe: { "filters": [ { "field": "example1", "operation": "like", "values": [ "Completed" ] }, { "field": "example2", "operation": "like", "values": [ "value1", "value2", "value3", ] } ] }  DF that i need:     example1 example2     Completed ["value1","value2","value3"]    

Answer: Try: dct = { "filters": [ {"field": "example1", "operation": "like", "values": ["Completed"]}, { "field": "example2", "operation": "like", "values": [ "value1", "value2", "value3", ], }, ] } df = pd.DataFrame( [ { f["field"]: f["values"][0] if len(f["values"]) == 1 else f["values"] for f in dct["filters"] } ] ) print(df)  Prints:  example1 example2 0 Completed [value1, value2, value3]
Question: import functools @functools.cache def get_some_results(): return results  Is there a way to notify the user of the function that the results they are getting are a cached version of the original for any other time they are calling the function?

Answer: This isn't a perfect approach, but you could use a custom decorator instead of @functools.cache which would then wrap your function with functools.cache and gather the cache stats before and after the call to determine if the lookup resulted in a cache hit. This was hastily thrown together but seems to work: def cache_notify(func): func = functools.cache(func) def notify_wrapper(*args, **kwargs): stats = func.cache_info() hits = stats.hits results = func(*args, **kwargs) stats = func.cache_info() if stats.hits > hits: print(f"NOTE: {func.__name__}() results were cached") return results return notify_wrapper  As an example of a simple function: @cache_notify def f(x): return -x print("calling f") print(f"f(1) returned {f(1)}") print("calling f again") print(f"f(1) returned {f(1)}")  Results: calling f f(1) returned -1 calling f again NOTE: f() results were cached f(1) returned -1  How to "notify the user" can be tailored as needed. Also note that it's possible for the cache stats to be a bit misleading in a multi-threaded environment; see Python lru_cache: how can currsize < misses < maxsize? for details.
Question: Given an input tuple, the goal is to produce a dictionary with some pre-defined keys, e.g. in this case, we have an add_header lambda and use the unpacking inside when calling the function. >>> z = (2, 1) >>> add_header = lambda x, y: {"EVEN": x, "ODD": y} >>> add_header(*z) {'EVEN': 2, 'ODD': 1}  My question is, is there way where the unpacking doesn't need to done when calling the add_header function? E.g. I can change avoid the lambda and do it in a normal function: >>> def add_header(w): ... x, y = w ... return {"EVEN": x, "ODD":y} ... >>> z = (2, 1) >>> add_header(z) {'EVEN': 2, 'ODD': 1}  Or I could not use the unpacking and use the index of the tuple, i.e. the w[0] and w[1]: >>> z = (2, 1) >>> add_header = lambda w: {"EVEN": w[0], "ODD": w[1]} >>> add_header(z) {'EVEN': 2, 'ODD': 1}  But is there some way to:  Use the lambda Don't explicitly use the indexing in the tuple Don't unpack with * when calling the add_header() function but it's okay to be inside the lambda function.  and still achieve the same {'EVEN': 2, 'ODD': 1} output given z = (2, 1) input?  I know this won't work but does something like this exist? z = (2, 1) add_header = lambda x, y from *w: {"EVEN": x, "ODD": y} add_header(z) 

Answer: You can try using dict() with zip(): z = (2, 1) add_header = lambda tpl: dict(zip(("EVEN", "ODD"), tpl)) print(add_header(z))  Prints: {'EVEN': 2, 'ODD': 1}
Question: I have the following numpy array with positive integers, in ascending order: import numpy as np arr = np.array([222, 225, 227, 228, 230, 232, 241, 243, 244, 245, 252, 253, 258])  I want to split it, into parts, where at each part, each number has maximum difference of 2 from the next one. So the following array should be split as: [[222], [225,227,228,230,232], [241, 243, 244, 245], [252, 253], [258]]  Any ideas how I can I achieve that ?

Answer: You can compute the diff, get the indices of differences above threshold with flatnonzero, and split with array_split: threshold = 2 out = np.array_split(arr, np.flatnonzero(np.diff(arr)>threshold)+1)  output: [array([222]), array([225, 227, 228, 230, 232]), array([241, 243, 244, 245]), array([252, 253]), array([258])]
Question: I have two dataframes that both have a column that can have the same number/value in it. One 'small df with ~300 rows (which is my leading file) and 1 df with ~ 5000 rows. I want to merge on 1 column but I cannot get the same amount of rows when I print the data. first (small) dataframe (left): import pandas as pd df1 = pd.read_excel('./file.xlsx') df1 = df.replace(' ', np.nan) df1.head() col1 row1 123456 row2 123457 row3 123458 row4 123459 row5 123450  second (big) df (right): import pandas as pd df2 = pd.read_excel('./file2.xlsx') df2 = df.replace(' ', np.nan) df2 col1 col2 row1 123456 hello1 row2 123457 hello2 row3 123458 hello3 row4 123459 hello4 row5 123450 hello4 row7 555555 street1 row8 666666 street1 row9 777777 street1  I tried: merged = pd.merge(left=df1, right=df2, how='inner', left_on='col1', right_on='col1') print("Orginele data", len(df1)) print("Merged data", len(df2))  When I print I get like 30k rows in the left df but I only want to see the rows used in the left df (~300 rows). Most of them are NaN's. I tried changing the 'how=' but that did not work. I also checked the post "Merging 101" but can't seem to figure this out. Expected result in left (small) dataframe:  col1 col2 row1 123456 hello1 row2 123457 hello2 row3 123458 hello3 row4 123459 hello4 row5 123450 hello4  Appreciate the help and effort. Thank you!

Answer: Try dataframe.join you can specify how='left which is by default import pandas as pd df = pd.DataFrame({"a": [0,0,1,1,2,2,2,]}) df2 = pd.DataFrame({"a": [0, 1,2,3,4,5,6,7,8,9], "b": list("abcdefghij")}) df.join(df2, on="a", lsuffix="df_a", rsuffix="df_b") # output adf_a adf_b b 0 0 0 a 1 0 0 a 2 1 1 b 3 1 1 b 4 2 2 c 5 2 2 c 6 2 2 c
Question: From the documentation page of Decimal, I thought that once we use decimal to compute, it'll be a correct result without any floating error. But when I try this equation from decimal import Decimal, getcontext getcontext().prec = 250 a = Decimal('6') b = Decimal('500000') b = a ** b print('prec: ' + str(getcontext().prec) + ', ', end='') print(b.ln() / a.ln())  It gives me different result!  I want to calculate the digit of 6**500000 in base-6 representation, so my expect result would be int(b.ln() / a.ln()) + 1, which should be 500001. However, when I set the prec to 250, it gives me the wrong result. How can I solve this? Also, if I want to output the result without the scientific notation (i.e. 5E+5), what should I do?

Answer: The Documentation clearly show the parameters of getcontext()  when you simply execute getcontext() it can show its built-in parameters. Context(prec=28, rounding=ROUND_HALF_EVEN, Emin=-999999, Emax=999999, capitals=1, clamp=0, flags=[], traps=[InvalidOperation, DivisionByZero, Overflow])  When you can change getcontext().prec = 250 then it can only overwrite prec value.  the main thing which can effect the result is rounding parameter right after the prec, It is built-in parameter rounding=ROUND_HALF_EVEN. I want to change this to None but it can show an error.  so it can clarify that no matter what we can do it must change even slightly because of rounding parameter. Note: your result may also effect because of other built-in parameters.
Question: I have a Github Action that runs unit tests on each Pull Request (PR). It effectively runs pytest. Seeing that we leverage the type hints introduced in PEP 484, I'd like a method like this to cause a PR check fail: def return_an_int() -> int: return 'not an int'  Is there a simple way to run such a "compilation" test (I know python is interpreted) for all the .py files in the project?

Answer: This is the kind of type checking that mypy does, so you could include running mypy in your Action workflow. If your module and its dependencies are already installed in your workflow, this step should do the trick:  - name: Run mypy run: | # needed if mypy isn't already installed pip install mypy # "mypy somedir" looks for and checks all Python files under somedir # I assume your code is in some source directory <src-dir> mypy <src-dir>  And that should be all you need, placed after the steps that install your code base.
Question: I am able to create quarterly and monthly PeriodIndex like so: idx = pd.PeriodIndex(year=[2000, 2001], quarter=[1,2], freq="Q") # quarterly idx = pd.PeriodIndex(year=[2000, 2001], month=[1,2], freq="M") # monthly  I would expect to be able to create a yearly PeriodIndex like so: idx = pd.PeriodIndex(year=[2000, 2001], freq="Y")  Instead this throws the following error: Traceback (most recent call last): File ".../script.py", line 3, in <module> idx = pd.PeriodIndex(year=[2000, 2001], freq="Y") File "/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/site-packages/pandas/core/indexes/period.py", line 250, in __new__ data, freq2 = PeriodArray._generate_range(None, None, None, freq, fields) File "/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/site-packages/pandas/core/arrays/period.py", line 316, in _generate_range subarr, freq = _range_from_fields(freq=freq, **fields) File "/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/site-packages/pandas/core/arrays/period.py", line 1160, in _range_from_fields ordinals.append(libperiod.period_ordinal(y, mth, d, h, mn, s, 0, 0, base)) File "pandas/_libs/tslibs/period.pyx", line 1109, in pandas._libs.tslibs.period.period_ordinal TypeError: an integer is required  It seems like something that should be very easy to do but yet I cannot understand what is going wrong. Can anybody help?

Answer: month and year are both required "fields" due to the current implementation (through pandas 1.5.1 at least). Most other field values will be configured with a default value, however, neither month or year will be defined if a value is not provided. Therefore, in this case, month will remain None which causes the error shown  TypeError: an integer is required  Here is a link to the relevant section of the source code where default values are defined. Omitting the month field results in [None, None] (in this case) which cannot be converted to a Periodindex.  A correct index can be built as follows. idx = pd.PeriodIndex(year=[2000, 2001], month=[1, 1], freq='Y')  Resulting in: PeriodIndex(['2000', '2001'], dtype='period[A-DEC]')  Depending on the number of years, it may also make sense to programmatically generate the list of months: years = [2000, 2001] idx = pd.PeriodIndex(year=years, month=[1] * len(years), freq='Y')   As an alternative, it may be easier to use to_datetime + to_period to create the Period index from a Datetime index instead (as it is already in a compatible form) pd.to_datetime([2000, 2001], format='%Y').to_period('Y')  Resulting in the same PeriodIndex: PeriodIndex(['2000', '2001'], dtype='period[A-DEC]')
Question: I've updated my version of Python to 3.11, but Terminal is printing different versions, depending on what command I enter. Entering python3 --version prints Python 3.9.13. Entering python --version prints Python 3.9.6. When I go to the actual Python framework, I can see that 3.11 is installed and is the current version, per the shortcut there. There are multiple versions of Python--Python 3.7, 3.8, etc.--there; perhaps this is the issue. I've looked into uninstalling some or all versions of Python, but I worry that will just make it worse--I'm not the most experienced programmer. I've also tried adding an alias to the .zshrc file per other posts, but it didn't work. I did save the file correctly, for what it's worth. Any advice is appreciated. macOS Ventura 13.0 Fix: I created a new user on my computer, which isn't a true fix, but it allowed me to bypass this issue (messy PATH) with relative ease.

Answer: I think you might be missing some foundational knowledge about how versions are selected from a terminal. You'll want to do some learning about PATH environment variable and how that relates to python versions. https://www.tutorialspoint.com/python/python_environment.htm#:~:text=The%20path%20is%20stored%20in,sensitive%3B%20Windows%20is%20not). The path that you're using is probably going to one version of python in your terminal but you have settings in whatever program you are using for the "python framework" is pointing to a different version. Also look a little into .bashrc and .zshrc files Python environments can be a bit annoying to manage at times. I'd recommend reading about pyenv and anaconda so that you can manage them. Good luck :) TLDR: Make sure your path for the terminal you are using is pointing to the correct location for the python version installation you want to use.
Question: I just upgraded from Python 3.7 to 3.9.14 and it now gives a variable not defined error. The same code works fine locally and remotely where Python 3.9.2 is installed but now locally it gives an error in Python 3.9.14 version. Below is the code: def check(url): result = None product = Product(url, user_agents) if product.is_connected(): result = product.parse() return result  if __name__ == '__main__': user_agents = [] with open('user-agents.txt', encoding='utf8') as f: user_agents = f.readlines() if len(links) > 0: print('Starting with the Pool count = ', PRODUCT_POOL_COUNT) with Pool(PRODUCT_POOL_COUNT) as p: result = p.map(check, links) result = list(filter(None, result)) # Remove Empty  Below is the error message: Traceback (most recent call last): File "/Users/Me/.pyenv/versions/3.9.14/lib/python3.9/multiprocessing/pool.py", line 125, in worker result = (True, func(*args, **kwds)) File "/Users/Me/.pyenv/versions/3.9.14/lib/python3.9/multiprocessing/pool.py", line 48, in mapstar return list(map(*args)) File "/Users/Me/Data/Clients/App/Etsy/products/parse_product.py", line 12, in check product = Product(url, user_agents) NameError: name 'user_agents' is not defined """ The above exception was the direct cause of the following exception: Traceback (most recent call last): File "/Users/Me/Data/Clients/App/Etsy/products/parse_product.py", line 126, in <module> result = p.map(check, links) File "/Users/Me/.pyenv/versions/3.9.14/lib/python3.9/multiprocessing/pool.py", line 364, in map return self._map_async(func, iterable, mapstar, chunksize).get() File "/Users/Me/.pyenv/versions/3.9.14/lib/python3.9/multiprocessing/pool.py", line 771, in get raise self._value NameError: name 'user_agents' is not defined 

Answer: You must be on OS-X. On that OS, Python changed across this versions the default method to spawn sub-processes - that also explains why it "works on Python 3.9 remotely": the remote deploy must be on a Linux or other Unix than MacOS - Bear with me: the default child-process creation method used to be "fork" for all Unixes - when "fork" is used, the new process is an exact copy of its parent, including all declared global variables - so the global variable user_agents exists and is visible in the target function. The new method for OS-X is "spawn": the new process starts-over all your project code, and re-execute all the lines, but for the lines guarded by the if __name__ == "__main__":  statement: in the child-processes, the variable __name__ contains the module actual name, as it is no longer the __main__ module of the running Python program (the original process is). Motivations for this change apart (they are easily searchable), the fix is simple: just declare your global variables outside of the guarded block: (...) user_agents = open("user-agents.txt", encoding="utf-8").readlines() if __name__ == '__main__': if len(links) > 0: print('Starting with the Pool count = ', PRODUCT_POOL_COUNT) with Pool(PRODUCT_POOL_COUNT) as p: result = p.map(check, links) result = list(filter(None, result)) # Remove Empty  (Also, no need to go through all of this of "with open" a file if you are reading it in a single glob)
Question: I want to split a column. If it has a letter (any letter) at the end, this will be the value for the second column. Otherwise, the second column should be null import pandas as pd data = pd.DataFrame({"data": ["0.00I", "0.01E", "99.99", "0.14F"]})  desired result:  a b 0 0.00 I 1 0.01 E 2 99.99 None 3 0.14 F 

Answer: You can use str.extract with the (\d+(?:\.\d+)?)(\D)? regex: out = data['data'].str.extract(r'(\d+(?:\.\d+)?)(\D)?').set_axis(['a', 'b'], axis=1)  Or, if you want to remove the original 'data' column while adding new columns in place: data[['a', 'b']] = data.pop('data').str.extract('(\d+(?:\.\d+)?)(\D)?')  output:  a b 0 0.00 I 1 0.01 E 2 99.99 NaN 3 0.14 F  regex demo (\d+(?:\.\d+)?) # capture a number (with optional decimal) (\D)? # optionally capture a non-digit
Question: I need to make a pyramid in python using recursion. Already made it, but I need help making it with recursion. def pyramid(n): for i in range(0, n): for j in range(0, i+1): print("* ",end="") print("\r") pyramid(5) 

Answer: Recursion = the repeated application of a recursive procedure. Code: def pyramid(n): if n==0: return else: pyramid(n-1) print("* "*n) n = 10 pyramid(n)  This just repeats the function until n = 0.
Question: I need to write a function called day_of_the_year that takes a month and day as input and returns the associated day of the year. Let the month by a number from 1 (representing January) to 12 (representing December). For example: day_of_the_year(1, 1) = 1 day_of_the_year(2, 1) = 32 day_of_the_year(3, 1) = 60  Use a loop to add up the full number of days for all months before the one you're interested in, then add in the remaining days. For example, to find the day of the year for March 5, add up the first two entries in days_per_month to get the total number of days in January and February, then add in 5 more days. So far, I have made a list: days_per_month = [31, 28, 31, 30, 31, 30, 31, 31, 30, 31, 30, 31]  This is my code so far: def day_of_the_year(month,day): days_per_month = [31, 28, 31, 30, 31, 30, 31, 31, 30, 31, 30, 31] for number in day_of_the_year: total = days_per_month[0] + days_per_month[1] + 5 return total  Is there something that I am missing?

Answer: Yes, you are exiting your loop on the first pass every time. What you should do is define the total variable outside of the for loop and then increment it on each iteration. You also only need to iterate to the month that is specified so use the range function to loop. And since day_of_the_year is the name of the function, it will cause an error if you try to put it in a for loop. Then once the loop has finished you can add the days to total and return it. def day_of_the_year(month,day): days_per_month = [31, 28, 31, 30, 31, 30, 31, 31, 30, 31, 30, 31] total = 0 for number in range(month - 1): total += days_per_month[number] return total + day print(day_of_the_year(1,1)) print(day_of_the_year(12,25))  output: 1 359  Michael's solution is the better solution to getting the answer, I just want to help you understand what you were missing and how to make it work.
Question: Does anyone can tell me how to use Numpy to reshape the Matrix [1,2,3,4] [5,6,7,8] [9,10,11,12] [13,14,15,16]  to [16,15,14,13] [12,11,10,9] [8,7,6,5] [4,3,2,1]  Thanks:) python 3.8 numpy 1.21.5 an example of my matrixs: [[ 1.92982258e+00 1.96782439e+00 2.00233048e-01 3.95128552e-01 4.21665915e-01 -1.10885581e-01 3.15967524e-01 1.86851601e-01] [ 5.82581567e-01 3.85242821e-01 6.52345512e-01 6.96774921e-01 4.46925274e-01 1.10208991e-01 -1.78544580e-02 2.63118328e-01] [ 1.18591189e-01 -8.87084649e-02 3.35701398e-01 3.81145692e-01 2.11622312e-02 3.10028567e-01 2.04480529e-01 4.45985566e-01] [ 5.59840625e-01 2.01962111e-01 5.34994738e-01 2.48421290e-01 2.42632687e-01 2.13238611e-01 3.96632085e-01 4.94549692e-01] [-7.69809051e-02 -3.00706661e-04 1.44790257e-01 3.49158021e-01 1.10096226e-01 2.03164938e-01 -3.45361600e-01 -3.33408510e-02] [ 2.33273192e-01 4.39144490e-01 -6.11938054e-02 -6.93128853e-02 -9.55871298e-02 -1.97338746e-02 -6.54788754e-02 2.81574199e-01] [ 6.61742595e-01 4.04149752e-01 2.33536310e-01 8.86332882e-02 -2.21808751e-01 -5.48789656e-03 5.49503834e-01 -1.22011728e-01] [-9.58502481e-03 2.36994437e-01 -1.28777627e-01 3.99751917e-01 -1.92452263e-02 -2.58119080e-01 3.40399940e-01 -2.20455571e-01]] 

Answer: You can rotate the matrix with numpy.rot90(). To get two rotations as your example, pass in k=2: import numpy as np a = np.array([ [1,2,3,4], [5,6,7,8], [9,10,11,12], [13,14,15,16], ]) np.rot90(a, k=2)  returning: array([[16, 15, 14, 13], [12, 11, 10, 9], [ 8, 7, 6, 5], [ 4, 3, 2, 1]])  Note the docs that say it returns a view of the original. This means the rotated matrix shares data with the original. Mutating one effects the other.
Question: Suppose you have df=pl.DataFrame( { "date":["2022-01-01", "2022-01-02"], "hroff":[5,2], "minoff":[1,2] }).with_columns(pl.col('date').str.to_date())  and you want to make a new column that adds the hour and min offsets to the date column. The only thing I saw was the dt.offset_by method. I made an extra column df=df.with_columns(pl.format('{}h{}m','hroff','minoff').alias('offset'))  and then tried df.with_columns(pl.col('date') \ .cast(pl.Datetime).dt.convert_time_zone('UTC') \ .dt.offset_by(pl.col('offset')).alias('newdate'))  but that doesn't work because dt.offset_by only takes a fixed string, not another column. What's the best way to do that?

Answer: Use pl.duration: import polars as pl df = pl.DataFrame({ "date": pl.Series(["2022-01-01", "2022-01-02"]).str.to_date(), "hroff": [5, 2], "minoff": [1, 2] }) print(df.select( pl.col("date") + pl.duration(hours=pl.col("hroff"), minutes=pl.col("minoff")) ))  shape: (2, 1) ┌─────────────────────┐ │ date │ │ --- │ │ datetime[μs] │ ╞═════════════════════╡ │ 2022-01-01 05:01:00 │ │ 2022-01-02 02:02:00 │ └─────────────────────┘