eshangj/stackoverflow_q_and_a_sample huggingface dataset:
Question: I have a line of code in polars that worked prior to my most recent update of the polars package to '0.19.0'. This example ran before: import polars as pl df = pl.DataFrame( { "a": [5, 6, 7, 8, 9], "b": [5, 6, 7, 8, 9], "c": [5, 6, 7, 8, None],}) cols_1 = ["a", "b"] cols_2 = ["c"] df = df.filter(pl.all(pl.col(cols_1 + cols_2).is_not_null()))  But now raises the error: ValueError: could not convert value 'Unknown' as a Literal 

Answer: Disclaimer. I am just summarising the comments of @jqurious and @keraion. As of the polars release 0.19.0, you'll need to use dedicated horizontal aggregation functions, such as all_horizontal or any_horizontal. import polars as pl df = pl.DataFrame({ "a": [5, 6, 7, 8, 9], "b": [5, 6, 7, 8, 9], "c": [5, 6, 7, 8, None] }) df.filter(pl.all_horizontal(pl.col(["a", "b", "c"]).is_not_null()))   The horizontal aggregations were introduced in release 0.18.7 and deprecated in 0.18.8.
Question: I have Excel Version 2309 (Build 16827.20000) through the Beta Channel of 365 Insider Program. The Python (Preview) icons do appear in the Formulas ribbon. I can also type Python code by =PY [tab] However, none of the code works, including the built-in samples. All I got is a #PYTHON! error. When I click show error message, it says "The Python interpreter returned the following error: Unexpected failure".  I have also tried each of the following codes like but all returns #PYTHON! 1+2 #Comment xl("A1:E5")  In addition to Excel, do I need to install something else to get Python works? Thanks in advance.

Answer: Same problem and finally solved. My 365 subscription is managed by my organization and I have two accounts: the first is my email with "company.com" domain, the second is my email with "company.onmicrosoft.com" domain. When I login with the first the PY function returns #PYTHON! error, while using the second with "onmicrosoft.com" domain it works. It seems a common issues for those having two accounts like me.
Question: I'm getting to grips with the Python SDK, having never used GraphQL before (but I'm familiar with the basic concept). I'm able to retrieve the odata_next_link value from responses, but I'm not sure how to use it. I note from here that:  You should include the entire URL in the @odata.nextLink property in your request for the next page of results. Depending on the API that the query is being performed against, the @odata.nextLink URL value will contain either a $skiptoken or a $skip query parameter. The URL also contains all the other query parameters present in the original request. Do not try to extract the $skiptoken or $skip value and use it in a different request.  However, I'm not sure how to include that URL in the next request. Currently, my queries look like response = await graph_client.groups.by_group_id(group_id).transitive_members.get() - I don't see an option there to change the base url. I thought I could do something like: query_params = GroupsRequestBuilder.GroupsRequestBuilderGetQueryParameters( skip_token = parse_qs(urlparse(response.odata_next_link).query)['$skipToken'][0] ) request_configuration = GroupsRequestBuilder.GroupsRequestBuilderGetRequestConfiguration( query_parameters=query_params ) response = await graph_client.[...].get(request_configuration)  but that reports GroupsRequestBuilder.GroupsRequestBuilderGetQueryParameters.__init__() got an unexpected keyword argument 'skip_token' (and similarly for if I try naming the parameter skiptoken or skipToken) Frustratingly, there's no code example here - but, based on those examples, I did search the repo for an Iterator - with no results.

Answer: We do not have a page iterator for the Python SDK yet, however there is added support for using a raw url for the request. In the case of pagination, a user can use the odata_next_link property value to make a fresh request using with_url and get items in the next page. Here's a working example:  from msgraph.generated.users.users_request_builder import UsersRequestBuilder async def get_users(): query_params = UsersRequestBuilder.UsersRequestBuilderGetQueryParameters( select=["id", "displayName", "createdDateTime"], top=5, ) request_config = UsersRequestBuilder.UsersRequestBuilderGetRequestConfiguration( query_parameters=query_params ) users_list = [] users = await client.users.get(request_configuration=request_config) for user in users.value: users_list.append(user) next_link = users.odata_next_link while next_link: users = await client.users.with_url(next_link).get() next_link = users.odata_next_link for user in users.value: users_list.append(user) print(len(users_list))
Question: I am writing out a compressed Parquet file from DataFrame as following: result_df.to_parquet("my-data.parquet", compression="zstd")  How can I instruct Pandas on the compression level of zstd coding?

Answer: Using pyarrow engine you can send compression_level in kwargs to to_parquet result_df.to_parquet(path, engine='pyarrow', compression='zstd', compression_level=1)  Test: import pandas as pd import pyarrow.parquet as pq path = 'my-data.parquet' result_df = pd.DataFrame({'a': range(100000)}) for i in range(10): # create the file result_df.to_parquet(path, engine='pyarrow', compression='zstd', compression_level=i) # get compressed file size metadata = pq.ParquetFile(path).metadata.row_group(0).column(0) print(f'compression level {i}: {metadata.total_compressed_size}')  Output: compression level 0: 346166 compression level 1: 309501 compression level 2: 309500 compression level 3: 346166 compression level 4: 355549 compression level 5: 381823 compression level 6: 310104 compression level 7: 310088 compression level 8: 308866 compression level 9: 308866
Question: I am using a windows machine and have installed wsl to be able to use Docker desktop. Of course the build failed and then I observed python3 and pip3 in the dockerfile. So I installed ubuntu and debian via wsl and then tried to run the app (docker-compose up). It still fails and throws the following error:  ERROR [test 3/5] RUN pip3 install daff==1.3.46 0.9s ------ > [test 3/5] RUN pip3 install daff==1.3.46: 0.812 error: externally-managed-environment 0.812 0.812 × This environment is externally managed 0.812 ╰─> To install Python packages system-wide, try apt install 0.812 python3-xyz, where xyz is the package you are trying to 0.812 install. 0.812 0.812 If you wish to install a non-Debian-packaged Python package, 0.812 create a virtual environment using python3 -m venv path/to/venv. 0.812 Then use path/to/venv/bin/python and path/to/venv/bin/pip. Make 0.812 sure you have python3-full installed. 0.812 0.812 If you wish to install a non-Debian packaged Python application, 0.812 it may be easiest to use pipx install xyz, which will manage a 0.812 virtual environment for you. Make sure you have pipx installed. 0.812 0.812 See /usr/share/doc/python3.11/README.venv for more information. 0.812 0.812 note: If you believe this is a mistake, please contact your Python installation or OS distribution provider. You can override this, at the risk of breaking your Python installation or OS, by passing --break-system-packages. 0.812 hint: See PEP 668 for the detailed specification. ------ failed to solve: process "/bin/sh -c pip3 install daff==1.3.46" did not complete successfully: exit code: 1  Here's the DockerFile: FROM postgres:12 RUN apt-get update && apt-get install -y \ python3 \ python3-pip RUN pip3 install daff==1.3.46 # Copy project files COPY . /app WORKDIR /app ENV PATH=$PATH:/app/bin  Any idea how to resolve the issue? Because I'm using docker, this How do I solve "error: externally-managed-environment" everytime I use pip3? doesn't help me.

Answer: This error occurs due to this: PEP0668. I found the resolution for local machines while going through this answer. Specifically for docker, using the pip install command with the flag --break-system-packages worked for me. In your script update the line: RUN pip3 install daff==1.3.46  with RUN pip3 install daff==1.3.46 --break-system-packages
Question: Given the following tree: ├── main.py └── my_module ├── a.py ├── b.py └── __init__.py  a.py: def f(): print('Hello World.')  b.py: from a import f def f2(): f() if __name__ == '__main__': f2()  main.py: from my_module.b import f2 if __name__ == '__main__': f2()  When I run b.py, "Hello World." is printed successfully. However, when I run main.py, I get the following error: Traceback (most recent call last): File "/home/user/main.py", line 1, in <module> from my_module.b import f2 File "/home/user/my_module/b.py", line 1, in <module> from a import f ModuleNotFoundError: No module named 'a'  I would expect the same output executing main.py and b.py

Answer: Because you have an __init__.py file in the my_module directory, Python sees the entirety of my_module as a package. That also means that all imports within my_module must be relative imports. This is easily fixed by changing your import in b.py to be a relative import. b.py from .a import f def f2(): f() if __name__ == '__main__': f2()  If you want to be able to run b.py as a separate script in addition to importing it as part of the the my_module package. You can make the imports conditionally relative or absolute based on whether __name__ == "__main__" if __name__ != '__main__': from .a import f def f2(): f() if __name__ == '__main__': from a import f f2()  Possibly a better alternative to checking if __name__ == '__main__': would be to use try/except try: from .a import f except ImportError: # If relative import failed (running as __main__), try absolute from a import f def f2(): f() if __name__ == '__main__': f2()  See also the definitive explanation of relative imports on SO
Question: I have the following DataFrame. df = pl.from_repr(""" ┌────────────┬───────────────┐ │ date ┆ maturity_date │ │ --- ┆ --- │ │ date ┆ date │ ╞════════════╪═══════════════╡ │ 2000-01-04 ┆ 2000-01-17 │ │ 2000-01-04 ┆ 2000-02-15 │ │ 2000-01-04 ┆ 2000-03-15 │ │ 2000-01-04 ┆ 2000-04-17 │ │ 2000-01-04 ┆ 2000-05-15 │ └────────────┴───────────────┘ """)  I'm trying to get the number of the workdays between date and maturity_date (not counting saturday and sunday) I'd also like to calculate diff days that use a given calendar like a trade date calendar of stock market which is different from a normal calendar. I use this date_ranges to count workdays, but it seems only a little faster than map_elements df.with_columns( pl.date_ranges("date", "maturity_date") .list.eval(pl.element().dt.weekday() <= 5) .list.count_matches(True) .alias("workdays_diff") # pl.concat_list("date", "maturity_date").map_elements(lambda x: get_work_days(x[0], x[1])) # .alias("workdays_diff") )  shape: (5, 3) ┌────────────┬───────────────┬───────────────┐ # ┌────────────────┐ │ date ┆ maturity_date ┆ workdays_diff │ # │ tradedate_diff │ │ --- ┆ --- ┆ --- │ # │ --- │ │ date ┆ date ┆ u32 │ # │ i64 │ ╞════════════╪═══════════════╪═══════════════╡ # ╞════════════════╡ │ 2000-01-04 ┆ 2000-01-17 ┆ 10 │ # │ 10 │ │ 2000-01-04 ┆ 2000-02-15 ┆ 31 │ # │ 21 │ │ 2000-01-04 ┆ 2000-03-15 ┆ 52 │ # │ 42 │ │ 2000-01-04 ┆ 2000-04-17 ┆ 75 │ # │ 65 │ │ 2000-01-04 ┆ 2000-05-15 ┆ 95 │ # │ 80 │ └────────────┴───────────────┴───────────────┘ # └────────────────┘  Is there a faster way? Is there also a way to calculate tradedate_diff?

Answer: @Dean MacGregor's excellent answer using pure Polars wasn't quite performant enough for my use case. Using Numpy's built in busday_count function turned out to be much faster in my case and can easily be converted back to a polars with pl.from_numpy. nyse_holidays=pl.Series([ date(2000,1,17), date(2000,2,21), date(2000,4,21), date(2000,5,29), date(2000,7,4), date(2000,9,4), date(2000,11,23), date(2000,12,25), ]) df = pl.DataFrame( { "business_days": pl.from_numpy(np.busday_count(data["date"], data["maturity_date"]),["business_days"])["business_days"], "trade_days": pl.from_numpy(np.busday_count(data["date"], data["maturity_date"], holidays=nyse_holidays),["trade_days"])["trade_days"], } ) shape: (5, 2) ┌───────────────┬────────────┐ │ business_days ┆ trade_days │ │ --- ┆ --- │ │ i64 ┆ i64 │ ╞═══════════════╪════════════╡ │ 9 ┆ 9 │ │ 30 ┆ 29 │ │ 51 ┆ 49 │ │ 74 ┆ 72 │ │ 94 ┆ 91 │ └───────────────┴────────────┘
Question: I am using pcolor to generate the following plot (code below). It has a colorbar in log scale and the x-values are in log-scale too. The problem is that the rectangles in this plot have different widths (I've put a red grid to show the rectangles better, suggestion of Trenton). Is there any way in which I can make sure the width of each rectangle is the same?  import matplotlib.pyplot as plt from matplotlib.colors import LogNorm import numpy as np # Generate Values x_values = np.geomspace(start=1, stop=1e-2, num=6) y_values = np.arange(start=0, stop=50, step=4, dtype=int) x_grid, y_grid = np.meshgrid(x_values, y_values) z_values = np.random.randn(len(y_values), len(x_values)) fig, ax = plt.subplots() im = ax.pcolor(x_grid, y_grid, z_values, norm=matplotlib.colors.LogNorm(), ec='r', lw=2) ax.set_xscale('log') fig.colorbar(im) plt.show() 

Answer: You need to specify the bin edges. Probably a better way to do this in numpy, but the idea is simple - transform to log space, get the bin edges by linear interpolation, and then transform back to normal space. import matplotlib.pyplot as plt from matplotlib.colors import LogNorm import numpy as np # Generate Values x_values = np.geomspace(start=1, stop=1e-2, num=6) y_values = np.arange(start=0, stop=50, step=4, dtype=int) # edges? logx = np.log10(x_values) edgex = np.hstack(( logx[:-1] - np.diff(logx) / 2, logx[-1] - np.diff(logx)[-1] / 2, logx[-1] + np.diff(logx)[-1] / 2)) edgex = 10**edgex edgey = np.hstack(( y_values[:-1] - np.diff(y_values) / 2, y_values[-1] - np.diff(y_values)[-1] / 2, y_values[-1] + np.diff(y_values)[-1] / 2)) np.random.seed(12345) z_values = np.random.randn(len(y_values), len(x_values)) fig, axs = plt.subplots(1, 2, layout='constrained') ax = axs[0] im = ax.pcolormesh(x_values, y_values, z_values, norm=LogNorm(), ec='r', lw=2) ax.set_xscale('log') ax.set_title('Linear gaps') ax.plot(x_values, 0 * x_values, 'dm') fig.colorbar(im) ax = axs[1] im = ax.pcolormesh(edgex, edgey, z_values, norm=LogNorm(), ec='r', lw=2) ax.plot(x_values, 0 * x_values, 'dm') ax.set_xscale('log') ax.set_title('Log gaps') fig.colorbar(im)
Question: Does setuptools support replacing the Manifest.in file, which specifies files that should only be included in the sdist distribution with a declaration in pyproject.toml?

Answer: Setuptools does not currently support replacing the MANIFEST.in with a declaration in pyproject.toml. There is also currently no specification for how to control the files included in an sdist. There are other build backends which do support this, by using their own tool subsection of pyproject.toml. For example, hatch uses [tool.hatch.build.targets.sdist] to control the sdist content, see File selection for more details about that.
Question: I have an array like so: arr = np.array([1, 2, 3, 4, -5, -6, 3, 5, 1, -2, 5, -1, -1, 10])  I want to get rid of all negative values, and split the array at each index where there was a negative value. The result should look like this: split_list = [[1, 2, 3, 4], [3, 5, 1], [5], [10]]  I know how to do this using list comprehension, but since the array can get quite large and I have to do the calculation many times, I want to find a solution using numpy. I found this https://www.geeksforgeeks.org/python-split-list-into-lists-by-particular-value/, which I can use to split the array where there are negative values, but I can't simultaneously remove them.

Answer: Note that instead of numpy, you could make use of itertools.groupby this way (though, judging on this: NumPy grouping using itertools.groupby performance, pure numpy will likely be more efficient): import numpy as np from itertools import groupby arr = np.array([1, 2, 3, 4, -5, -6, 3, 5, 1, -2, 5, -1, -1, 10]) split_list = [list(group) for key, group in groupby(arr, key=lambda x:x>=0) if key] # [[1, 2, 3, 4], [3, 5, 1], [5], [10]]
Question: I have two pandas DataFrame, with the same structure. Dataframe B is a subset of DataFrame A. I want to filter DataFrame B, only if the Price value appears in DataFrame A, or it is within 1% of a value in DataFrame A. For example, even if the exact price is not present, I want to keep the value if there is a row in A with a price +/- 1%. DataFrame A: index price 0 20.23 1 10.34 2 5.28 3 12.25 4 12.32  DataFrame B: index price 0 0.23 1 10.34 2 5.26  Desired Result of filtering: index price 0 10.34 1 5.26  import pandas as pd dfA = pd.DataFrame({'index': [0, 1, 2, 3, 4], 'price': [20.23, 10.34, 5.28, 12.25, 12.32]}) dfB = pd.DataFrame({'index': [0, 1, 2], 'price': [0.23, 10.34, 5.26]})  The following will only give me the exact matches. dfB[dfB['price'].isin(dfA['price'])] 

Answer: Here is another take, using .merge_asof (note: dfA and dfB needs to be sorted by price): dfA = dfA.sort_values("price") dfB = dfB.sort_values("price") x = pd.merge_asof(dfB, dfA, on="price", direction="nearest")  Creates x:  index_x price index_y 0 0 0.23 2 1 2 5.26 2 2 1 10.34 1   Then calculating the percentage difference: x["diff"] = np.abs(x["price"] - dfA.loc[x["index_y"], "price"].values) print(x[x["diff"] <= (x["price"] / 100)])  Prints:  index_x price index_y diff 1 2 5.26 2 0.02 2 1 10.34 1 0.00   The complete code: dfA = dfA.sort_values("price") dfB = dfB.sort_values("price") x = pd.merge_asof(dfB, dfA, on="price", direction="nearest") x["diff"] = np.abs(x["price"] - dfA.loc[x["index_y"], "price"].values) x = x[x["diff"] <= (x["price"] / 100)] x = x[["index_x", "price"]].set_index("index_x").rename_axis(index=None).sort_index() print(x)  Prints:  price 1 10.34 2 5.26
Question: I am trying to convert a for loop into an iterator using yield, but I have failed in my attempts. I don't understand exactly why the yield isn't giving me the expected output. Does anyone know what the problem is? Attempt at using yield: def iteration_order(dimensions): for dim in range(dimensions): order = [0, dim, 0] yield order for j in range(6): sgn = 1 if j % 2 == 0 else -1 idx = j % 3 for _ in range(dim if j < 5 else dim-1): order[idx] += sgn yield order  print(list(iteration_order(2)) >>> [[0, 0, 0], [0, 1, 1], [0, 1, 1], [0, 1, 1], [0, 1, 1], [0, 1, 1], [0, 1, 1]]  The code as it should work (when not using yield): def iteration_order(dimensions): full_order = [] for dim in range(dimensions): order = [[0, dim, 0]] for j in range(6): sgn = 1 if j % 2 == 0 else -1 idx = j % 3 for _ in range(dim if j < 5 else dim-1): nxt = list(order[-1]) nxt[idx] += sgn order.append(nxt) full_order.extend(order) return full_order  print(iteration_order(2)) >>> [[0, 0, 0], [0, 1, 0], [1, 1, 0], [1, 0, 0], [1, 0, 1], [0, 0, 1], [0, 1, 1]] 

Answer: The problem you see is because you are using that same list for everything. You might yield it with different value, but the generator still has reference to that list and it modifies it, giving you weird output. If you add .copy() for each yield, they will be unique lists and will behave as expected: def iteration_order(dimensions): for dim in range(dimensions): order = [0, dim, 0] yield order.copy() for j in range(6): sgn = 1 if j % 2 == 0 else -1 idx = j % 3 for _ in range(dim if j < 5 else dim-1): order[idx] += sgn yield order.copy() print(list(iteration_order(2))) # [[0, 0, 0], [0, 1, 0], [1, 1, 0], [1, 0, 0], [1, 0, 1], [0, 0, 1], [0, 1, 1]]
Question: I'm trying to calculate the likelihood of successfully guessing a password in Python. For example, if we take a 10 character lowercase password (26**10 possible passwords) and we can make 1 billion guesses a second, we can calculate the probability of successfully guessing the password in one hour with: from scipy.stats import binom (1 - binom.pmf(k=0, n=1000000000*3600, p=1 / 26**10))  Which gives us the the result of 0.02525515384826793 (i.e, 2.5%). However, this doesn't work as we increase the length of the password (or more strictly, as p gets closer to zero). For instance if we increase the length of the password to 12 characters: from scipy.stats import binom (1 - binom.pmf(k=0, n=1000000000*3600, p=1 / 26**12))  Then the returned value is just 0.0, which is incorrect - presumably due to the float rounding down to zero at some point. How can I calculate this to get a more precise answer?  Edit: this was with SciPy 1.10.1. Testing on the latest version (1.11.2 at time of writing) gave the correct value - so it looks like this was an issue with the older version.

Answer: I think this is about the limitations of floating-point arithmetic which I also often face (i.e, float overflow). you can use the mpmath library, which provides arbitrary-precision arithmetic: from mpmath import mp mp.dps = 50 # Set the desired precision (adjust as needed) total_passwords = 26**12 guesses_per_second = 1000000000 seconds_in_an_hour = 3600 p = mp.mpf(1) / mp.mpf(total_passwords) n = mp.mpf(guesses_per_second) * mp.mpf(seconds_in_an_hour) probability = 1 - mp.power(1 - p, n) print(float(probability))  Extra-Note: Floating-point numbers lose precision even when you are just working with such seemingly harmless numbers like 0.2 or 76.5. You should be extra careful when working with a large amount of floating-point operations over the same data as errors may build up rather quickly. I hope this can help somehow.
Question: Of course we have: "1,2,3".split(",") # ["1", "2", "3"] "1".split(",") # ["1"]  but also this, which is sometimes problematic in some situations (*): "".split(",") # [""]  Is there a built-in way (maybe with a parameter, or a specific function) to have: "".split(",", allow_empty=True) # []  ? This would (sometimes) make sense: the input is empty, so the output list should be empty. (*) Example situation: for element in s.split(","): print(f"we have the element {element}") # works for s = "1, 2, 3" # works for s = "1" # doesn't work for s = "" => the loop should be empty 

Answer: Not to my knowledge but you could use a regex with re.findall: import re re.findall(r'[^,]+', '1,2,3') # ['1', '2', '3'] re.findall(r'[^,]+', '1') # ['1'] re.findall(r'[^,]+', '') # []  Note that this would also discard empty strings within the input string: re.findall(r'[^,]+', '1,,3') # ['1', '3']