eshangj/stackoverflow_q_and_a_sample huggingface dataset:
Question: Polars noob, given an m x n Polars dataframe df and a 1 x n Polars dataframe of scalars, I want to divide each column in df by the corresponding scalar in the other frame. import numpy as np import polars as pl cols = list('abc') df = pl.DataFrame(np.linspace(1, 9, 9).reshape(3, 3), schema=cols) scalars = pl.DataFrame(np.linspace(1, 3, 3)[:, None], schema=cols)  In [13]: df Out[13]: shape: (3, 3) ┌─────┬─────┬─────┐ │ a ┆ b ┆ c │ │ --- ┆ --- ┆ --- │ │ f64 ┆ f64 ┆ f64 │ ╞═════╪═════╪═════╡ │ 1.0 ┆ 2.0 ┆ 3.0 │ │ 4.0 ┆ 5.0 ┆ 6.0 │ │ 7.0 ┆ 8.0 ┆ 9.0 │ └─────┴─────┴─────┘ In [14]: scalars Out[14]: shape: (1, 3) ┌─────┬─────┬─────┐ │ a ┆ b ┆ c │ │ --- ┆ --- ┆ --- │ │ f64 ┆ f64 ┆ f64 │ ╞═════╪═════╪═════╡ │ 1.0 ┆ 2.0 ┆ 3.0 │ └─────┴─────┴─────┘  I can accomplish this easily in Pandas as shown below by delegating to NumPy broadcasting, but was wondering what the best way to do this is without going back and forth between Polars / Pandas representations. In [16]: df.to_pandas() / scalars.to_numpy() Out[16]: a b c 0 1.0 1.0 1.0 1 4.0 2.5 2.0 2 7.0 4.0 3.0  I found this similar question where the scalar constant is already a row in the original frame, but don't see how to leverage a row from another frame. Best I can come up with thus far is combining the frames and doing some... nasty looking things :D In [31]: (pl.concat([df, scalars]) ...: .with_columns(pl.all() / pl.all().tail(1)) ...: .head(-1)) Out[31]: shape: (3, 3) ┌─────┬─────┬─────┐ │ a ┆ b ┆ c │ │ --- ┆ --- ┆ --- │ │ f64 ┆ f64 ┆ f64 │ ╞═════╪═════╪═════╡ │ 1.0 ┆ 1.0 ┆ 1.0 │ │ 4.0 ┆ 2.5 ┆ 2.0 │ │ 7.0 ┆ 4.0 ┆ 3.0 │ └─────┴─────┴─────┘ 

Answer: I think you found out a very unique/interesting and clever solution. Consider also just iterating over columns: df.select(column / scalars[column.name] for column in df.iter_columns())  or df.select(pl.col(k) / scalars[k] for k in df.columns)  or df.with_columns(pl.col(k).truediv(scalars[k]) for k in df.columns)
Question: I want to encode the non-zero binary events with integer numbers. Following is a demo table: import polars as pl df = pl.DataFrame( { "event": [0, 1, 1, 0], "foo": [1, 2, 3, 4], "boo": [2, 3, 4, 5], } )  The expected output is achieved by: df = df.with_row_index() events = df.select(pl.col(["index", "event"])).filter(pl.col("event") == 1).with_row_index("event_id").drop("event") df = df.join(events, on="index", how="left") out: shape: (4, 5) ┌───────┬───────┬─────┬─────┬──────────┐ │ index ┆ event ┆ foo ┆ boo ┆ event_id │ │ --- ┆ --- ┆ --- ┆ --- ┆ --- │ │ u32 ┆ i64 ┆ i64 ┆ i64 ┆ u32 │ ╞═══════╪═══════╪═════╪═════╪══════════╡ │ 0 ┆ 0 ┆ 1 ┆ 2 ┆ null │ │ 1 ┆ 1 ┆ 2 ┆ 3 ┆ 0 │ │ 2 ┆ 1 ┆ 3 ┆ 4 ┆ 1 │ │ 3 ┆ 0 ┆ 4 ┆ 5 ┆ null │ └───────┴───────┴─────┴─────┴──────────┘  I want to get the expeceted output by chaining the expressions: ( df .with_row_index() .join( df .select(pl.col(["index", "event"])) .filter(pl.col("event") == 1) .with_row_index("event_id") .drop("event"), on="index", how="left", ) )  However, the expressions within the .join() expression does not seem to have added index column from the df.with_row_index() operation: ColumnNotFoundError: index Error originated just after this operation: DF ["event", "foo", "boo"]; PROJECT */3 COLUMNS; SELECTION: "None" 

Answer: While the solution using the walrus operator works. It is probably more idiomatic and cleaner to use a pl.when().then() construct in conjunction with pl.int_range() to create the event_id. ( df .with_columns( pl.when(pl.col("event") == 1) .then(pl.int_range(pl.len())) .over("event") .alias("event_id") ) )  shape: (4, 4) ┌───────┬─────┬─────┬──────────┐ │ event ┆ foo ┆ boo ┆ event_id │ │ --- ┆ --- ┆ --- ┆ --- │ │ i64 ┆ i64 ┆ i64 ┆ i64 │ ╞═══════╪═════╪═════╪══════════╡ │ 0 ┆ 1 ┆ 2 ┆ null │ │ 1 ┆ 2 ┆ 3 ┆ 0 │ │ 1 ┆ 3 ┆ 4 ┆ 1 │ │ 0 ┆ 4 ┆ 5 ┆ null │ └───────┴─────┴─────┴──────────┘
Question: I want to get a single value from a pandas DataFrame more efficiently. This is how I do it now: # import pandas import pandas as pd # set up the dataframe df = pd.DataFrame({'col1':['a','a','b'],'col2':[10,20,20],'col3':[100.0,200.0,300.0]}) # STEP 1 - filter down to a single row with loc row_of_interest = df.loc[(df['col1'] == 'a') & (df['col2'] > 11)] # STEP 2 - specify the column of interest column_and_row_of_interest = row_of_interest['col3'] # STEP 3 - get the value out of dataframe format using to_list and list indexing value_of_interest = column_and_row_of_interest.to_list()[0]  And of course, I can do steps 1-3 in a single line of code like this: value_of_interest = df.loc[(df['col1'] == 'a') & (df['col2'] > 11)]['col3'].to_list()[0]  I imagine STEP 1 and STEP 2 might be unavoidable, but STEP 3 feels clunky. Is there a better way to get a value out of a DataFrame with a shape of (1,1) than using .to_list()[0] ?

Answer: You can do: print(column_and_row_of_interest.squeeze())  OR: print(column_and_row_of_interest.iat[0])   This prints: 200.0
Question: Sorry if my question doesn't make a lot of sense. I don't have much experience in python. I have some code that looks like: import polars as pl from typing import NamedTuple class Event(NamedTuple): name: str description: str def event_table(num) -> list[Event]: events = [] for i in range(5): events.append(Event("name", "description")) return events def pretty_string(events: list[Event]) -> str: pretty = "" for event in events: pretty += f"{event.name}: {event.description}\n" return pretty # This does work print(pretty_string(event_table(5))) # But then it doesn't work if I have my `list[Event]` in a dataframe data = {"events": [0, 1, 2, 3, 4]} df = pl.DataFrame(data).select(events=pl.col("events").map_elements(event_table)) # This doesn't work pretty_df = df.select(events=pl.col("events").map_elements(pretty_string)) print(pretty_df) # Neither does this print(pretty_string(df["events"][0]))  It fails with error: Traceback (most recent call last): File "path/to/script.py", line 32, in <module> pretty_df = df.select(events=pl.col("events").map_elements(pretty_string)) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "path/to/.venv/lib/python3.11/site-packages/polars/dataframe/frame.py", line 8116, in select return self.lazy().select(*exprs, **named_exprs).collect(_eager=True) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "path/to/.venv/lib/python3.11/site-packages/polars/lazyframe/frame.py", line 1934, in collect return wrap_df(ldf.collect()) ^^^^^^^^^^^^^ polars.exceptions.ComputeError: AttributeError: 'dict' object has no attribute 'name'  Looks like my list[Event] is no longer that inside the df. I am not sure how to go about getting this to work.

Answer: You can keep the Event objects by passing return_dtype=pl.Object df.select(pl.col("events").map_elements(event_table))  shape: (5, 1) ┌───────────────────────────────────┐ │ events │ │ --- │ │ list[struct[2]] │ ╞═══════════════════════════════════╡ │ [{"name","description"}, {"name"… │ │ [{"name","description"}, {"name"… │ │ [{"name","description"}, {"name"… │ │ [{"name","description"}, {"name"… │ │ [{"name","description"}, {"name"… │ └───────────────────────────────────┘  df.select(pl.col("events").map_elements(event_table, return_dtype=pl.Object))  shape: (5, 1) ┌───────────────────────────────────┐ │ events │ │ --- │ │ object │ ╞═══════════════════════════════════╡ │ [Event(name='name', description=… │ │ [Event(name='name', description=… │ │ [Event(name='name', description=… │ │ [Event(name='name', description=… │ │ [Event(name='name', description=… │ └───────────────────────────────────┘
Question: Trying to avoid typing issues I often run into the same problem. E.g. I have a function x that very rarily returns value None, all other times it returns int.  def x(i: int) -> Union[int, None]: if i == 0: return return i def test(i: int): a = x(i) # typing issue: *= not supported for types int | None and int a *= 25  x used very often in the codebase and most of the time i was already checked a hundred times that x(i) will indeed return int and not None. Using it as int right away creates typing warnings - e.g. you can't multiply possible None value. What's best practice for that case? Ideas I considered:  There is no real sense to check it for None with if a is None: return as it's already known. a *= 25 # type: ignore will make a an Unknown type. a = x(i) # type: int will make the warning go away. But will create a new warning "int | None cannot be assigned to int" a = cast(int, x(i)), haven't tested it much yet.  I usually end up changing return type of x to just int, adding ignore in return # type: ignore and mention in the docstring that it can return None, it helps avoiding contaminating the entire codebase with type warnings. Is this the best approach? def x(i: int) -> int: """might also return `None`""" if i == 0: return # type: ignore return i 

Answer: This might be a case where an exception is better than a return statement you never expect to be reached. def x(i: int) -> int: if i == 0: raise ValueError("didn't expect i==0") return i def test(i: int): try: a = x(i) except ValueError: pass a *= 25  Code that is confident it has sufficiently validated the argument to x can omit the try statement. Statically speaking, this is accurate: if x returns, it is guaranteed to return an int. (Whether it will return is another question.)  Ideally, you could define a refinement type like NonZeroInt, and turn i == 0 into a type error, rather than a value error. # Made-up special form RefinementType obeys # # isinstance(x, RefinementType[T, p]) == isinstance(x, T) and p(x) NonZeroInt = RefinementType[int, lambda x: x != 0] def x(i: NonZeroInt) -> int: return i x(0) # error: Argument 1 to "x" has incompatible type "int"; expected "NonZeroInt" [arg-type] i: int = 0 x(i) # same error j: NonZeroInt = 0 # error: Incompatible types in assignment (expression has type "int", variable has type "NonZeroInt") [assignment] x(j) # OK k: NonZeroInt = 3 # OK x(k) # OK
Question: I regularly work with Python files that are provided to me as templates, which use an indentation of 2. However, I personally prefer working with an indentation width of 4, which is what I've set in my .vimrc. However, because of the indentation-sensitivity of Python, the typical gg=G way to fix indentation to my preferred indentation width does not work well at all to convert a file to my preferred indentation. Furthermore, just leaving the indentation at a width of 2 screws up with my tabstop settings. To fix this, what I'd like to do is have some system to convert a 2-space indented file into a 4-space indented file. I don't know what the easiest way to do this would be, but I'm thinking to vmap <Tab> to increase the tab width of the selected lines by 1, and vmap <S-Tab> to decrease the width by 1. I don't know if there is some built-in method to do this, or if this would require some find/replace rule, but I think having some way to fix the indentation to my preferred width would be useful, given that the autoindent is not smart enough to fix this properly.

Answer: In vim: " Convert 2-spaces indent to tabs :set noexpandtab tabstop=2 :retab! " Convert tabs to 4-spaces indent :set expandtab tabstop=4 :retab
Question: I recently found a bug in my code that I was able to fix by replacing np.divide(x, y) with x / y. I was under the impression that np.divide(x, y) was equivalent to x / y (it says as much in the numpy documentation). Is this a bug in numpy or is it expected behaviour? As I said my immediate issue is solved so I'm not too worried about finding a fix, I am more curious to understand what's going on. import numpy as np x1 = np.array([[281], [15831], [30280], [975], [313], [739], [252], [10364], [21480], [1447], [315], [772], [95], [2710], [7408], [215], [111], [158], [0], [88], [21], [661], [0], [0], [0], [5], [4], [0], [12], [0], [0], [50], [28], [0], [0], [272]]) x2 = np.array([[499], [6315], [33800], [580], [208], [464], [384], [3127], [19596], [2319], [218], [1740], [217], [411], [4250], [223], [406], [267], [2], [0], [16], [0], [0], [0], [0], [8], [3], [0], [18], [0], [1], [0], [41], [0], [0], [0]]) x3 = np.array([[507], [6180], [34005], [555], [200], [451], [390], [3024], [19492], [2425], [211], [1848], [223], [396], [4097], [224], [406], [282], [2], [0], [16], [0], [0], [0], [0], [8], [3], [0], [19], [0], [2], [0], [45], [0], [0], [0]]) x4 = np.array([[507], [6178], [34017], [554], [200], [451], [391], [3022], [19486], [2439], [210], [1865], [223], [396], [4089], [224], [406], [284], [2], [0], [16], [0], [0], [0], [0], [8], [3], [0], [19], [0], [2], [0], [46], [0], [0], [0]]) not_zero = (x1 + x2) != 0 x = np.divide(2*(x1 - x2)**2, x1 + x2, where=not_zero) r = (2*(x1[not_zero] - x2[not_zero])**2) / (x1[not_zero] + x2[not_zero]) print("n1 =",x.max(),"\tt1 =", r.max()) not_zero = (x2 + x3) != 0 x = np.divide(2*(x2 - x3)**2, x2 + x3, where=not_zero) r = (2*(x2[not_zero] - x3[not_zero])**2) / (x2[not_zero] + x3[not_zero]) print("n2 =",x.max(),"\tt2 =", r.max()) not_zero = (x3 + x4) != 0 x = np.divide(2*(x3 - x4)**2, x3 + x4, where=not_zero) r = (2*(x3[not_zero] - x4[not_zero])**2) / (x3[not_zero] + x4[not_zero]) print("n3 =",x.max(),"\tt3 =", r.max())  Output: n1 = 8177.933351395286 t1 = 8177.933351395286 n2 = 873842.0 t2 = 6.501672240802676 n3 = 1322.0 t3 = 0.15566927013196877  Python version: 3.7.6 Numpy version: 1.17.0

Answer: The parameter where = mask without the parameter out is somewhat dangerous. Without a target for the output, the function builds an np.empty array of the appropriate shape, and then replaces some subset of the empty array with the output data. But np.empty isn't, well, empty. It's just a random memory location that hasn't been initialized (so it still has any garbage data that existed in that memory block before). So where mask = False, your output will be that leftover random garbage. If that memory block happens to have binary garbage that can be encoded into a number bigger than the rest of your data, it end up being your max value. You can mask out the garbage using not_zero as a mask again: x = np.divide(2*(x1 - x2)**2, x1 + x2, where=not_zero)[not_zero]  or initialize your output array yourself: x = np.zeros_like(x1) np.divide(2*(x1 - x2)**2, x1 + x2, where = not_zero, out = x)
Question:   Koda Ulaşılamıyor -> Code is unreachable  Visual Studio code is graying out my code and saying it is unreachable after I used pd.concat(). The IDE seems to run smoothly but it's disturbing and I want my colorful editor back. How do I disable the editor graying out my code without changing the current language?

Answer: This is a bug currently existing in pandas-stubs. The matching overload of concat in pandas-stubs currently returns Never. According to this suggestion in Pylance github, you could work around the pandas-stubs issue by commenting out the Never overload in ...\.vscode\extensions\ms-python.vscode-pylance-2024.3.1\dist\bundled\stubs\pandas\core\reshape\concat.pyi. @overload def concat( objs: Iterable[None] | Mapping[HashableT1, None], *, axis: Axis = ..., join: Literal["inner", "outer"] = ..., ignore_index: bool = ..., keys: Iterable[HashableT2] = ..., levels: Sequence[list[HashableT3] | tuple[HashableT3, ...]] = ..., names: list[HashableT4] = ..., verify_integrity: bool = ..., sort: bool = ..., copy: bool = ..., ) -> Never: ...
Question: I often work in very small projects which do not have config file. How do I use ruff in place of isort to sort the imports? I know that the following command is roughly equivalent to black: ruff format .  The format command do not sort the imports. How do I do that?

Answer: According to the documentation:  Currently, the Ruff formatter does not sort imports. In order to both sort imports and format, call the Ruff linter and then the formatter:  ruff check --select I --fix . ruff format .
Question: My google-fu is failing me. I have a simple dataframe that looks like this:    Sample Subject Person Place Thing     1-1 Janet      1-1 Boston      1-1 Hat      1-2 Chris      1-2 Austin      1-2 Scarf       I want the values in the subject column to move into their appropriate column so that I end up with something like this:    Sample Subject Person Place Thing     1-1 Janet Janet Boston Hat   1-2 Chris Chris Austin Scarf    I've looked at pivot and transpose, but those don't seem right. Any ideas would be appreciated! :)

Answer: If the groups are sorted and the pattern is always the same (no missing values), then reshape with numpy: cols = ['Person', 'Place', 'Thing'] out = df.loc[::len(cols), ['Sample']].reset_index(drop=True) out[cols] = df['Subject'].to_numpy().reshape(-1, len(cols))  For a more generic approach, only assuming that the categories are always in the same order within a group, identify the position per group with groupby.cumcount and map the names, then pivot: order = ['Person', 'Place', 'Thing'] out = (df.assign(col=df.groupby('Sample').cumcount() .map(dict(enumerate(order)))) .pivot(index='Sample', columns='col', values='Subject') .reset_index().rename_axis(columns=None) )  Variant with rename: order = ['Person', 'Place', 'Thing'] out = (df.assign(col=df.groupby('Sample').cumcount()) .pivot(index='Sample', columns='col', values='Subject') .rename(columns=dict(enumerate(order))) .reset_index().rename_axis(columns=None) )  Output:  Sample Person Place Thing 0 1-1 Janet Boston Hat 1 1-2 Chris Austin Scarf  Finally, if you really want the "Subject" column, insert it: out.insert(1, 'Subject', out['Person']) print(out) Sample Subject Person Place Thing 0 1-1 Janet Janet Boston Hat 1 1-2 Chris Chris Austin Scarf  timings If you can use the numpy approach, it's more strict on the input but much faster:
Question: I need to generate a heatmap with the average coverage of positions within a bin from a determined number of bins, regardless of the number of bases in a transcriptome within each bin. In other words, if I want to have 10 bins, for one transcriptome, it may have 1000 bases to distribute among 10 bins, and another may have 2445 bases to distribute among 10 bins. The problem is that in my coverage file, there are gaps that don't fall into any bin. For example, if I want 5 bins over 10 positions, I'll have: (0,2], (2,4], (4,6], (6,8], (8,10]. If my positions with coverage are 1, 5, 5, 5, 7, 7, 10, the bin "(2,4]" will be hidden, thus not appearing in the heatmap. What I want is for these bins without coverage to be filled with 0s so that they appear in the heatmap. I'm using python with pandas, seaborn and matplot.pyplot libraries In the image below, the first line is edges positions of my bins, and the dataframe is what bins have coverage: enter image description here Input example: chr17 1 1 chr17 5 1 chr17 5 2 chr17 5 2 chr17 7 1 chr17 7 5 chr17 10 1  Problem:  chr data_bin avg chr17 (0,2] 1 chr17 (4,6] 1.66 chr17 (4,6] 1.66 chr17 (4,6] 1.66 chr17 (6,8] 3 chr17 (6,8] 3 chr17 (8,10] 1  Expected:  chr data_bin avg chr17 (0,2] 1 **chr17 (2,4] 0** chr17 (4,6] 1.66 chr17 (4,6] 1.66 chr17 (4,6] 1.66 chr17 (6,8] 3 chr17 (6,8] 3 chr17 (8,10] 1  The function I am using is: def bins_calculator(path_txt:str, start:int,end:int): column_names =["chr", "pos", "cov"] data = pd.read_csv(path_txt, names = column_names, sep = '\t') step = int((end - start) / 10) n_bins = [start + i * step for i in range(11)] n_bins[-1] = end data["data_bin"] = pd.cut(data["pos"], bins = n_bins) data["avg"] = data.groupby("data_bin", observed = False)["cov"].transform("mean") filtered_data = data[["chr","data_bin","avg"]].drop_duplicates("data_bin") return filtered_data  Any questions about this problem, please let me know in the comments :)

Answer: IIUC you can use .merge to merge the missing categories, then fill any NaNs with values you want: df["data_bin"] = pd.cut(df["pos"], range(0, 12, 2)) df = pd.merge( df, df["data_bin"].cat.categories.to_frame(), left_on="data_bin", right_on=0, how="outer", )[["chr", "data_bin", "cov"]] df["chr"] = df["chr"].ffill().bfill() df["cov"] = df["cov"].fillna(0) df["avg"] = df.groupby("data_bin")["cov"].transform("mean") print(df)  Prints:  chr data_bin cov avg 0 chr17 (0.0, 2.0] 1.0 1.000000 1 chr17 (2.0, 4.0] 0.0 0.000000 2 chr17 (4.0, 6.0] 1.0 1.666667 3 chr17 (4.0, 6.0] 2.0 1.666667 4 chr17 (4.0, 6.0] 2.0 1.666667 5 chr17 (6.0, 8.0] 1.0 3.000000 6 chr17 (6.0, 8.0] 5.0 3.000000 7 chr17 (8.0, 10.0] 1.0 1.000000
Question: Is there something like row_number, or a row_count expression in Polars? Something like the Polars with_row_index as an expression, but not just as a data frame method. I want to rank a given customer's order number in a day using the over window function, but can't come up with a solution so far. pl.count().over('ID Client')) gives me the total number of interactions that day. Now I want to have them ordered... In SQL I believe this works simply with the RANK() or ROW_NUMBER() window functions. Can anyone help with it?

Answer: You might be searching for pl.int_range, potentially combined with pl.Expr.over. import polars as pl df = pl.DataFrame({ "group": ["A", "A", "A", "B", "B", "C"], }) df.with_columns(pl.int_range(pl.len()).over("group").alias("index"))  shape: (6, 2) ┌───────┬───────┐ │ group ┆ index │ │ --- ┆ --- │ │ str ┆ i64 │ ╞═══════╪═══════╡ │ A ┆ 0 │ │ A ┆ 1 │ │ A ┆ 2 │ │ B ┆ 0 │ │ B ┆ 1 │ │ C ┆ 0 │ └───────┴───────┘
Question: I am attempting to decode a byte sequence into a Unicode string from various types of files, such as .exe, .dll, and .deb files, using the pefile library in Python. However, I sometimes encounter Unicode decoding errors. How can I handle these errors effectively? Here's the relevant code snippet: import pefile def get_section_addresses(file_path): section_addresses = {} pe = pefile.PE(file_path) for section in pe.sections: section_addresses[section.Name.decode().strip('\x00')] = section.VirtualAddress return section_addresses section_addresses = get_section_addresses('D:/Binary/file/rufus.exe') for name, address in section_addresses.items(): print(f"{name}:{address:08X}")  I'm utilizing pefile to parse Portable Executable (PE) files, extracting section names and their corresponding virtual addresses. However, during the decoding of section names, I sometimes encounter Unicode decoding errors.

Answer: I've implemented error handling using nested try-except blocks  try: pe = pefile.PE(file_path) for section in pe.sections: try: name = section.Name.decode().strip('\x00') except UnicodeDecodeError: name = "Undecodable" section_addresses[name] = section.VirtualAddress except pefile.PEFormatError: print(f"Error: {file_path} is not a valid PE file.") return section_addresses