eshangj/stackoverflow_q_and_a_sample huggingface dataset:
Question: I have the following code and output. Code. import polars as pl df = pl.DataFrame({ 'type': ['A', 'O', 'B', 'O'], 'id': ['CASH', 'ORB.A123', 'CHECK', 'OTC.BV32'] }) df.with_columns(sub_id=pl.when(pl.col('type') == 'O').then(pl.col('id').str.split('.')).otherwise(None))  Output. shape: (4, 3) ┌──────┬──────────┬─────────────────┐ │ type ┆ id ┆ sub_id │ │ --- ┆ --- ┆ --- │ │ str ┆ str ┆ list[str] │ ╞══════╪══════════╪═════════════════╡ │ A ┆ CASH ┆ null │ │ O ┆ ORB.A123 ┆ ["ORB", "A123"] │ │ B ┆ CHECK ┆ null │ │ O ┆ OTC.BV32 ┆ ["OTC", "BV32"] │ └──────┴──────────┴─────────────────┘  Now, how would I extract the n-th element (or in this case, the last element) of each list? Especially, the expected output is the following. shape: (4, 3) ┌──────┬──────────┬────────────┐ │ type ┆ id ┆ sub_id │ │ --- ┆ --- ┆ --- │ │ str ┆ str ┆ str │ ╞══════╪══════════╪════════════╡ │ A ┆ CASH ┆ null │ │ O ┆ ORB.A123 ┆ "A123" │ │ B ┆ CHECK ┆ null │ │ O ┆ OTC.BV32 ┆ "BV32" │ └──────┴──────────┴────────────┘ 

Answer: You can simply append .list.last() to select the last element of each list. Alternatively, there exists .list.get() to get list elements by index. import polars as pl df = pl.DataFrame({ 'type': ['A', 'O', 'B', 'O'], 'id': ['CASH', 'ORB.A123', 'CHECK', 'OTC.BV32'] }) df.with_columns( sub_id=pl.when( pl.col('type') == 'O' ).then( pl.col('id').str.split('.').list.last() ) )  shape: (4, 3) ┌──────┬──────────┬────────┐ │ type ┆ id ┆ sub_id │ │ --- ┆ --- ┆ --- │ │ str ┆ str ┆ str │ ╞══════╪══════════╪════════╡ │ A ┆ CASH ┆ null │ │ O ┆ ORB.A123 ┆ A123 │ │ B ┆ CHECK ┆ null │ │ O ┆ OTC.BV32 ┆ BV32 │ └──────┴──────────┴────────┘  Note that I've dropped .otherwise(None) as this is the default behaviour of an if-then-otherwise expression.
Question: Suppose I have a dataframe in which I want to replace a non-regex substring consisting only of characters (i.e. a-z, A-Z) and/or digits (i.e. 0-9) via pd.Series.str.replace. The docs state that this function is equivalent to str.replace or re.sub(), depending on the regex argument (default False). Apart from most likely being overkill, are there any downsides to consider if the function was called with regex=True for non-regex replacements (e.g. performance)? If so, which ones? Of course, I am not suggesting using the function in this way. Example: Replace 'Elephant' in the below dataframe. import pandas as pd data = {'Animal_Name': ['Elephant African', 'Elephant Asian', 'Elephant Indian', 'Elephant Borneo', 'Elephant Sumatran']} df = pd.DataFrame(data) df = df['Animal_Name'].str.replace('Elephant', 'Tiger', regex=True) 

Answer: Special characters! Using regular expressions with plain words is generally fine (aside from efficiency concerns), there will however be an issue when you have special characters. This is an often overlooked issue and I've seen many people not understanding why their str.replace failed. Pandas even changed the default regex=True to regex=False, and the original reason for that (#GH24804) was that str.replace('.', '') would remove all characters, which is expected if you know regex, but not at all if you don't. For example, let's try to replace 1.5 with 2.3 and the $ currency by £: df = pd.DataFrame({'input': ['This item costs $1.5.', 'We need 125 units.']}) df['out1'] = df['input'].str.replace('1.5', '2.3', regex=False) df['out1_regex'] = df['input'].str.replace('1.5', '2.3', regex=True) df['out2'] = df['input'].str.replace('$', '£', regex=False) df['out2_regex'] = df['input'].str.replace('$', '£', regex=True)  Output:  input out1 out1_regex \ 0 This item costs $1.5. This item costs $2.3. This item costs $2.3. 1 We need 125 units. We need 125 units. We need 2.3 units. out2 out2_regex 0 This item costs £1.5. This item costs $1.5.£ 1 We need 125 units. We need 125 units.£  Since . and $ have a special meaning in a regex, those cannot be used as is and should have been escaped (1\.5 / \$), which can be done programmatically with re.escape. How does str.replace decide to use a regex or a plain string operation? a pure python replacement will be used if:  regex=False pat is a string (passing a compiled regex with regex=False will trigger a ValueError) case is not False no flags are set repl is not a callable  In all other cases, re.sub will be used. The code that does this is is core/strings/object_array.py:  def _str_replace( self, pat: str | re.Pattern, repl: str | Callable, n: int = -1, case: bool = True, flags: int = 0, regex: bool = True, ): if case is False: # add case flag, if provided flags |= re.IGNORECASE if regex or flags or callable(repl): if not isinstance(pat, re.Pattern): if regex is False: pat = re.escape(pat) pat = re.compile(pat, flags=flags) n = n if n >= 0 else 0 f = lambda x: pat.sub(repl=repl, string=x, count=n) else: f = lambda x: x.replace(pat, repl, n)  Efficiency Considering a pattern without special characters, regex=True is about 6 times slower than regex=False in the linear regime:
Question: When I create a date range in pandas, I often use the periods argument. Something like this: pd.date_range(start='1/1/2018', periods=8)  DatetimeIndex(['2018-01-01', '2018-01-02', '2018-01-03', '2018-01-04', '2018-01-05', '2018-01-06', '2018-01-07', '2018-01-08'], dtype='datetime64[ns]', freq='D')  What would be the equivalent way in polars? I am missing the periods input parameter in pl.date_range. Having said that, there's probably an easy and clever solution ;-)

Answer: Adding a periods argument has been an open feature request for a while now. Until the request has been implemented, you can make start an expression and create end by offsetting start by the desired number of periods (using pl.Expr.dt.offset_by). start = pl.lit("1/1/2018").str.to_date() pl.date_range(start=start, end=start.dt.offset_by("7d"), eager=True)  shape: (8,) Series: 'literal' [date] [ 2018-01-01 2018-01-02 2018-01-03 2018-01-04 2018-01-05 2018-01-06 2018-01-07 2018-01-08 ]
Question: Is it possible to merge the two "replace" methods in the example below into one? The intention is for scalability. import pandas as pd custom_groups = {"odd": [1, 3, 5], "even": [2, 4]} s = pd.Series([1, 2, 3, 4, 5]) s.replace(custom_groups["odd"], "odd").replace(custom_groups["even"], "even") 

Answer: Most pythonic/efficient approach: reverse the dictionary with a dictionary comprehension and replace once: d = {v:k for k, l in custom_groups.items() for v in l} s.replace(d)  Output: 0 odd 1 even 2 odd 3 even 4 odd dtype: object  Intermediate d: {1: 'odd', 3: 'odd', 5: 'odd', 2: 'even', 4: 'even'}  timings This is ~2x faster than using a Series+explode on small datasets, and tends to the same speed for large datasets. ## OP's example # dictionary comprehension 745 µs ± 31.9 µs per loop (mean ± std. dev. of 7 runs, 1000 loops each) # Series explode 1.58 ms ± 213 µs per loop (mean ± std. dev. of 7 runs, 1000 loops each) ## 1000 keys dictionary / 10 items per list / 10_000 items Series # dictionary comprehension 163 ms ± 6.61 ms per loop (mean ± std. dev. of 7 runs, 10 loops each) # Series explode 203 ms ± 64.4 ms per loop (mean ± std. dev. of 7 runs, 1 loop each)  generalization If you have to do this often with different dictionaries, you can even use a function to invert the dictionary def invert(d): return {v:k for k, l in custom_groups.items() for v in (l if isinstance(l, list) else [l])} s.replace(invert(custom_groups))
Question: Here is a minimal example: import numpy as np np.set_printoptions(linewidth=1000, precision=3) # First attempt fails to limit the printed precision of x x = np.array([None]) x[0] = 1/3 print(x) # Second attempt succeeds x = [None] x[0] = 1/3 x = np.array(x) print(x)  Running this script yields [0.3333333333333333] [0.333]  Why does the "First attempt" above fail to limit the printed precision of x while the second attempt succeeds?

Answer: When running: x = np.array([None]) x[0] = 1/3 print(x)  x is an object array (that contains python floats), not an array with a float dtype like your second attempt: array([0.3333333333333333], dtype=object)  This ignores the print options. You can reproduce this simply with: print(np.array([1/3], dtype=object), np.array([1/3]))  Output: [0.3333333333333333] [0.333] As a workaround, convert your array to float: print(x.astype(np.float64)) # [0.333]
Question: I'd like to count the number of unique values when combining several columns at once. My idea so far was to use pl.struct(...).n_unique(), which works fine when I consider missing values as a unique value: import polars as pl df = pl.DataFrame({ "x": ["a", "a", "b", "b"], "y": [1, 1, 2, None], }) df.with_columns(foo=pl.struct("x", "y").n_unique())  shape: (4, 3) ┌─────┬──────┬─────┐ │ x ┆ y ┆ foo │ │ --- ┆ --- ┆ --- │ │ str ┆ i64 ┆ u32 │ ╞═════╪══════╪═════╡ │ a ┆ 1 ┆ 3 │ │ a ┆ 1 ┆ 3 │ │ b ┆ 2 ┆ 3 │ │ b ┆ null ┆ 3 │ └─────┴──────┴─────┘  However, sometimes I want to exclude a combination from the count if it contains any number of missing values. In the example above, I'd like foo to be 2. However, using .drop_nulls() before counting doesn't work and produces the same output as above. df.with_columns(foo=pl.struct("x", "y").drop_nulls().n_unique())  Is there a way to do this using only Polars expressions?

Answer: pl.Expr.drop_nulls does not drop the row as the entirety of the struct is indeed not null. To still achieve the desired result, you can filter out all rows which contain a null values in any of the columns of interest using pl.Expr.filter. ( df .with_columns( foo=pl.struct("x", "y").filter( ~pl.any_horizontal(pl.col("x", "y").is_null()) ).n_unique() ) )  shape: (4, 3) ┌─────┬──────┬─────┐ │ x ┆ y ┆ foo │ │ --- ┆ --- ┆ --- │ │ str ┆ i64 ┆ u32 │ ╞═════╪══════╪═════╡ │ a ┆ 1 ┆ 2 │ │ a ┆ 1 ┆ 2 │ │ b ┆ 2 ┆ 2 │ │ b ┆ null ┆ 2 │ └─────┴──────┴─────┘
Question: In django's LOGGING configuration for the builtin django.db.backends it states that: "For performance reasons, SQL logging is only enabled when settings.DEBUG is set to True, regardless of the logging level or handlers that are installed." As a result the following LOGGING configuration, which is correctly set up to issue debug level logs showing DB queries, will NOT output the messages I need:  DEBUG = False LOGGING = { "version": 1, "disable_existing_loggers": True, "root": {"handlers": [ "gcp_structured_logging"]}, "handlers": { "gcp_structured_logging": { "level": "DEBUG", "class": "django_gcp.logging.GoogleStructuredLogsHandler", } }, "loggers": { 'django.db.backends': { 'handlers': ["gcp_structured_logging"], 'level': 'DEBUG', 'propagate': True, }, }, }  This is preventing me from activating this logging in production, where of course I'm not going to durn on DEBUG=True in my settings but where I need to log exactly this information. Ironically, I need this in order to debug a performance issue (I plan to run this for a short time in production and cat my logs so I can set up a realistic scenario for a load test and some benchmarking on the database). How can I override django's override so that sql queries get logged as I intend? NOTE willeM_ Van Onsem's answer as accepted is correct (because it totally is) but it's worth noting that in the end, I came across a library called django-silk. Whilst not an answer to this question directly, silk actually covers the capability I was trying to build for myself when I found this peculiarity of how the db logging works. Perhaps someone else trying to achieve the same thing will make good use of it.

Answer: Fortunately we can override this. Indeed, by setting the .force_debug_cursor of the connection to True, for example in one of the AppConfigs (any app config) exists: # my_app/apps.py from django.apps import AppConfig from django.db import connection class MyAppConfig(AppConfig): name = 'my_app' def ready(self): connection.force_debug_cursor = True This works because Django decides whether to log with [GitHub]:  @property def queries_logged(self): return self.force_debug_cursor or settings.DEBUG   Another option is to work with the CaptureQueriesContext [GitHub], this is a context processor that sets .force_debug_cursor to true in the scope of the context. So if you know where to look for the slow query, you can use: from django.db import connection from django.test.utils import with CaptureQueriesContext(connection): list(MyModel.objects.all()) # sample query
Question: I have a pl.DataFrame with a start_date and end_date column. I need to compute the difference between those two columns and add new columns representing the result in days, weeks, months and years. I would be fine to get an approximate result, meaning dividing the days by 7 / 30 / 365. My problem is to convert the duration[ns] type into an integer type. import datetime import polars as pl df = pl.DataFrame( {"start_date": datetime.date(2024, 1, 1), "end_date": datetime.date(2024, 7, 31)} ) df = df.with_columns((pl.col("end_date") - pl.col("start_date")).alias("days")) print(df) shape: (1, 3) ┌────────────┬────────────┬──────────────┐ │ start_date ┆ end_date ┆ days │ │ --- ┆ --- ┆ --- │ │ date ┆ date ┆ duration[ms] │ ╞════════════╪════════════╪══════════════╡ │ 2024-01-01 ┆ 2024-07-31 ┆ 212d │ └────────────┴────────────┴──────────────┘ 

Answer: you can use dt.total_days() to extract hours from Duration datatype: df.with_columns( days = (pl.col.end_date - pl.col.start_date).total_days() ) ┌────────────┬────────────┬──────┐ │ start_date ┆ end_date ┆ days │ │ --- ┆ --- ┆ --- │ │ date ┆ date ┆ i64 │ ╞════════════╪════════════╪══════╡ │ 2024-01-01 ┆ 2024-07-31 ┆ 212 │ └────────────┴────────────┴──────┘
Question: I have a dataframe with multiple columns containing unix epochs. In my example I only use 2 of 13 columns I have. I'd like to cast all those columns to a datetime with UTC timezone in a single call to with_columns(). df = pl.from_repr(""" ┌─────┬────────────┬────────────┬────────────┐ │ id ┆ start_date ┆ end_date ┆ cancelable │ │ --- ┆ --- ┆ --- ┆ --- │ │ i64 ┆ i64 ┆ i64 ┆ bool │ ╞═════╪════════════╪════════════╪════════════╡ │ 1 ┆ 1566637530 ┆ 1566628686 ┆ true │ │ 2 ┆ 1561372720 ┆ 1561358079 ┆ true │ │ 3 ┆ 1561374780 ┆ 1561358135 ┆ false │ │ 4 ┆ 1558714718 ┆ 1556188225 ┆ false │ │ 5 ┆ 1558715044 ┆ 1558427697 ┆ true │ └─────┴────────────┴────────────┴────────────┘ """)  Polars provides the user with pl.from_epoch. However, I didn't find a way to apply it to multiple columns as once. Expected result: shape: (5, 4) ┌─────┬─────────────────────┬─────────────────────┬────────────┐ │ id ┆ start_date ┆ end_date ┆ cancelable │ │ --- ┆ --- ┆ --- ┆ --- │ │ i64 ┆ datetime[μs] ┆ datetime[μs] ┆ bool │ ╞═════╪═════════════════════╪═════════════════════╪════════════╡ │ 1 ┆ 2019-08-24 09:05:30 ┆ 2019-08-24 06:38:06 ┆ true │ │ 2 ┆ 2019-06-24 10:38:40 ┆ 2019-06-24 06:34:39 ┆ true │ │ 3 ┆ 2019-06-24 11:13:00 ┆ 2019-06-24 06:35:35 ┆ false │ │ 4 ┆ 2019-05-24 16:18:38 ┆ 2019-04-25 10:30:25 ┆ false │ │ 5 ┆ 2019-05-24 16:24:04 ┆ 2019-05-21 08:34:57 ┆ true │ └─────┴─────────────────────┴─────────────────────┴────────────┘  So far, my code looks as follows. columns_epoch_to_timestamp: list[str] = [ "start_date", "end_date", ] df = df.with_columns(pl.col(*columns_epoch_to_timestamp)) 

Answer: pl.from_epoch() accepts a single column name, but also Expr objects.  column: str | Expr | Series | Sequence[int]  pl.col() can select multiple columns, which we can pass directly: df.with_columns(pl.from_epoch(pl.col("start_date", "end_date")))  shape: (5, 4) ┌─────┬─────────────────────┬─────────────────────┬────────────┐ │ id ┆ start_date ┆ end_date ┆ cancelable │ │ --- ┆ --- ┆ --- ┆ --- │ │ i64 ┆ datetime[μs] ┆ datetime[μs] ┆ bool │ ╞═════╪═════════════════════╪═════════════════════╪════════════╡ │ 1 ┆ 2019-08-24 09:05:30 ┆ 2019-08-24 06:38:06 ┆ true │ │ 2 ┆ 2019-06-24 10:38:40 ┆ 2019-06-24 06:34:39 ┆ true │ │ 3 ┆ 2019-06-24 11:13:00 ┆ 2019-06-24 06:35:35 ┆ false │ │ 4 ┆ 2019-05-24 16:18:38 ┆ 2019-04-25 10:30:25 ┆ false │ │ 5 ┆ 2019-05-24 16:24:04 ┆ 2019-05-21 08:34:57 ┆ true │ └─────┴─────────────────────┴─────────────────────┴────────────┘
Question: I would like to split array into chunks, sum the values in each chunk, and return the result as another array. The chunks can have different sizes. This can be naively done by using numpy split function like this def split_sum(a: np.ndarray, breakpoints: np.ndarray) -> np.ndarray: return np.array([np.sum(subarr) for subarr in np.split(a, breakpoints)])  However, this still uses a python for-loop and is thus inefficient for large arrays. Is there a faster way?

Answer: You wouldn't really split an array in numpy unless this is the last step. Numpy can handle your operation natively with numpy.add.reduceat (a minor difference with your function is how the breakpoints are defined, you will need to prepend 0 with reduceat): arr = np.arange(20) breakpoints = np.array([2, 5, 10, 12]) def split_sum(a: np.ndarray, breakpoints: np.ndarray) -> np.ndarray: return np.array([np.sum(subarr) for subarr in np.split(a, breakpoints)]) split_sum(arr, breakpoints) # array([ 1, 9, 35, 21, 124]) np.add.reduceat(arr, np.r_[0, breakpoints]) # array([ 1, 9, 35, 21, 124])
Question: I am looking at someones code and this is what they wrote from financetoolkit import Toolkit API_KEY = "FINANCIAL_MODELING_PREP_API_KEY" companies = Toolkit(["AAPL", "MSFT", "GOOGL", "AMZN"], api_key=API_KEY, start_date="2005-01-01") income_statement_growth = companies.get_income_statement(growth=True) display(income_statement_growth.loc[:, "Revenue", :])  Essentially what this code does is it returns the revenue value for a couple of companies starting from 2005 until present day. What I am confused about is income_statement_growth.loc[:, "Revenue", :] why is there three arguments? I dont understand what the third colon is doing in this code All the documentation I read about .loc states that it takes two arguments, one for the row and one for the column, so I am a bit confused how it is able to take three and what the function of the third colon is.

Answer: The return value of companies.get_income_statement(growth=True) is a pandas DataFrame with a multi-index. The columns are indexed by period ('2019', '2020', etc.) and the rows are indexed by a combination of company ticker and data item (e.g. ('AAPT', 'Revenue')). You could access a single element like this: print(income_statement_growth['2020'][('AAPL', 'Revenue')])  And to select the 'Revenue' for all tickers and all periods, you use .loc: revenues = income_statement_growth.loc[:, 'Revenue', :]  For a simple dataframe, you world normally see two arguments for .loc[] but since this is a multi-index, this needs three arguments.
Question: I have an array A (of size m x n), and a percentage p in [0,1]. I need to produce an m x n boolean array B, with True in in the (i,j) entry if A[i,j] is in p^{th} quantile of the column A[:,j]. Here is the code I have used so far. import numpy as np m = 200 n = 300 A = np.random.rand(m, n) p = 0.3 quant_levels = np.zeros(n) for i in range(n): quant_levels[i] = np.quantile(A[:,i],p) B = np.array(A >= quant_levels) 

Answer: I'm not sure it's much faster but you should at least be aware that numpy.quantile has an axis keyword argument so you can compute all the quantiles with one command: quant_levels = np.quantile(A, p, axis=0) B = (A >= quant_levels)
Question: I'm using ruff as a replacement to black formatter but I wanna keep the diff at minimum. I'm noticing that it automatically inserts a newline between the module-level docstring and the first import statement. For example, given this code: """base api extension.""" import abc from typing import List, Optional, Type  After running: ruff format file.py --diff  It gives me this: @@ -1,4 +1,5 @@ """base api extension.""" + import abc from typing import List, Optional, Type  If I format the file, the output is like this: """base api extension.""" import abc from typing import List, Optional, Type  I wanna keep the original formatting without adding that newline. I couldn't find any settings I could use to ignore this. Is there a way to configure ruff to prevent this behaviour? Thank you! My pyproject.toml before adding ruff: [tool.black] line_length = 120 include = '\.py$' [tool.isort] multi_line_output = 3 include_trailing_comma = true force_grid_wrap = 0 line_length = 120 profile = "black"  After adding ruff: [tool.ruff] line-length = 120  Context: Python: 3.11.9 Ruff Version: 0.5.7 

Answer: If you update your black version you'll find the same issue. This formatting change is internally called module_docstring_newlines:  Black version 24.1.0 (26 Jan 2024) introduced the new "2024 stable style" which includes this change (which was previously only in "preview style") - see Black's changelogs (look for #3932). Ruff version 0.3.0 (29 Feb 2024) thus chose to also introduce these changes in their 2024.2 stable style - see Ruff's changelogs (look for #8283).  This comment on the Ruff pull request for this change confirms that there is unfortunately no config option to prevent this behaviour. If you read through the linked issues and pull request discussions for Black and Ruff you'll see it mentioned in both cases that the formatting change often affects lots of files, but (ignoring this large caveat) it was otherwise considered uncontroversial.
Question: While reading json from files, using polars give exception when it has empty json. But it is not the same in pandas. I have some files with only key and not value. Is polars still primitive and not matured like pandas? which package to use? Polars from io import StringIO import polars as pl print(pl.read_json(StringIO('{"key":{}}')))  gives exception pyo3_runtime.PanicException: called `Result::unwrap()` on an `Err` value: ComputeError(ErrString("a StructArray must contain at least one field"))  Pandas from io import StringIO import pandas as pd print(pd.read_json(StringIO('{"key":{}}')))  works without exception, gives output as Empty DataFrame Columns: [key] Index: [] 

Answer: Pandas is mature and Polars is comparatively a new library. There is an issue exists, which is same as yours, refer to this bug link, Polars Issue Link Either you can use exception handling with polars for this scenario or go with pandas. Update: The issue is closed now, you it is working in Polars
Question: a = np.array([101,105,90,102,90,10,50]) b = np.array([99,110,85,110,85,90,60]) expected result = np.array([2,5,5,8,5,20,10])  How can I find minimum absolute difference value between elements using just numpy operations; with modulus of 100 if two values are across 100.

Answer: The answer by Derek Roberts with numpy.minimum is almost correct. However, since the input values can be greater than 100, they should first be rescaled to 0-100 with %100 (mod). I'm adding an extra pair of values to demonstrate this: a = np.array([101,105,90,102,90,10,50,1001]) b = np.array([99,110,85,110,85,90,60,2]) x = abs(a%100-b%100) np.minimum(x, 100-x)  Generic computation: M = 100 x = abs(a%M-b%M) np.minimum(x, M-x)  Output: array([ 2, 5, 5, 8, 5, 20, 10, 1])