eshangj/stackoverflow_q_and_a_sample huggingface dataset:
Question: In a Python script I make a call to a SOAP service which returns an XML reply where the elements have a namespace prefix, let's say <ns0:foo xmlns:ns0="SOME-URI"> <ns0:bar>abc</ns0:bar> </ns0:foo>  I can extract the content of ns0:bar with the method call doc.getElementsByTagName('ns0:bar')  However, the name ns0 is only a local variable so to speak (it's not mentioned in the schema) and might as well have been named flubber or you_should_not_care. What is the proper way to extract the content of a namespaced element without relying on it having a specific name? In my case the prefix was indeed changed in the SOAP service which resulted in a parse failure.

Answer: Namespace support is needed if searching by element name doc.getElementsByTagNameNS('SOME-URI','bar')  If using a package with namespace support like lxml tree.findall('{http://schemas.xmlsoap.org/soap/envelope/}Body')  or by local name  tree.xpath('//*[local-name()="bar"]'  lxml example from lxml import etree tree = etree.parse("/home/lmc/tmp/soap.xml") tree.xpath('//*[local-name()="Company"]')  Result [<Element {http://example.com}Company at 0x7f0959fb3fc0>]
Question: Intention: I am trying to create a unit test for a complex class, where a lot of values are randomly generated by using random.random(). To create a unit test, I want to use mock.patch to set fixed values for random.random(), to receive always same values (same configuration) and then I can run my test which must have always same result. Problem: I need to patch function random() from random library with values depending on the module. In my understanding, mock.patch('modul1.random.random', return_value=1) should only influence the modul1 and no other random() functions in other modules. The same for the modul2: modul1.py: import random def function(): return random.random()  modul2.py: import random def function(): return random.random()  The unit_test: def test_function(): with mock.patch('modul1.random.random', return_value=1), \ mock.patch('modul2.random.random', return_value=0): val1 = modul1.function() val2 = modul2.function() assert not val1 == val2  Expectation: val1 = 1 and val2 = 0, therefore passed Reality: assert not 0 == 0 PythonCodebase/tests/test_patient.py:55: AssertionError

Answer: There is only function to patch, random.random, and it's shared by both modules. The best you can do is use side_effect to provide two values for it to return, one per call, but that requires you to know the order in which modul1.function and modul2.function will be called, and that may not be predictable. Better would be to modify the two modules to use their own names to refer to random.random; then you can patch those two names separately. modul1.py: from random import random def function(): return random()  modul2.py: from random import random def function(): return random()  The unit_test: def test_function(): with mock.patch('modul1.random', return_value=1), \ mock.patch('modul2.random', return_value=0): val1 = modul1.function() val2 = modul2.function() assert not val1 == val2
Question: I have an array of indexes: test_idxs = np.array([4, 2, 7, 5])  I also have an array of values (which is longer): test_vals = np.array([13, 19, 31, 6, 21, 45, 98, 131, 11])  So I want to get an array with the length of the array of indexes, but with values from the array of values in the order of the array of indexes. In other words, I want to get something like this: array([21, 31, 131, 45])  I know how to do this in a loop, but I don't know how to achieve this using Numpy tools.

Answer: This is actually extremely simple with numpy, just index your test_vals array with test_idx (integer array indexing): out = test_vals[test_idxs]  Output: array([ 21, 31, 131, 45])  Note that this requires the indices to be valid. If you have indices that could be too high you would need to handle them explicitly. Example: test_idxs = np.array([4, 2, 9, 5]) test_vals = np.array([13, 19, 31, 6, 21, 45, 98, 131, 11]) out = np.where(test_idxs < len(test_vals), test_vals[np.clip(test_idxs, 0, len(test_vals)-1)], np.nan)  Output: array([21., 31., nan, 45.])
Question: How does plotly add shapes to figures with multiple subplots and what best practices are around that? Let's take the following example: from plotly.subplots import make_subplots fig = make_subplots(rows=2, cols=1, shared_xaxes=True) fig.add_vrect(x0=1, x1=2, row=1, col=1, opacity=0.5, fillcolor="grey") fig.add_scatter(x=[1,3], y=[3,4], row=1, col=1) fig.add_scatter(x=[2,2], y=[3,4], row=2, col=1)   If we add_vrect at the end, the rectangle is visualized as I would expect. fig = make_subplots(rows=2, cols=1, shared_xaxes=True) fig.add_scatter(x=[1,3], y=[3,4], row=1, col=1) fig.add_scatter(x=[2,2], y=[3,4], row=2, col=1) fig.add_vrect(x0=1, x1=2, row=1, col=1, opacity=0.5, fillcolor="grey")   Now, when I move away from the dummy example to a more complex plot (3 subplots, multiple y axes, logarithmic scaling, datetime x axis), adding the rectangles last does not help either. I don't manage to visualize them for two of the three subplots. Thus, I'm trying to better understand how plotly handles this under the hood. From what I have gathered so far, the rectangles are shapes and thus, not part of figure.data, but figure.layout. In the above dummy examples, the shapes are only added to the layout in the second take. Why? Is it more advisable to use fig.add_shape(type="rect") when working with more complex plots? Or should I give up and just manually wrangle with fig.layout.shapes instead of using the function calls? Examples are made with plotly 5.15.0.

Answer: The problem with shapes in subplots arises from how plotly assigns axis references (xref and yref). add_vrect automatically maps shapes to subplots using row and col which can fail in complex layouts. For more control, use add_shape. For a rectangle that always stretches vertically across the entire plot, similar to vrect, define it with reference to a secondary y-axis. This ensures it remains fixed from top to bottom even when zooming. Here is an example: import plotly.graph_objects as go from plotly.subplots import make_subplots import pandas as pd data = { "time": pd.date_range("2023-01-01", periods=10, freq="D"), "value1": [10, 15, 20, 25, 30, 35, 40, 45, 50, 55], "value2": [5, 10, 15, 10, 5, 10, 15, 10, 5, 10], } df = pd.DataFrame(data) fig = make_subplots(rows=2, cols=1, shared_xaxes=True, specs=[[{"secondary_y": True}], [{}]]) fig.add_shape( type="rect", x0="2023-01-03", x1="2023-01-06", y0=0, y1=1, xref="x1", yref="y2", secondary_y=True, fillcolor="blue", opacity=0.3, line_width=0, ) fig.add_scatter(x=df["time"], y=df["value1"], row=1, col=1, name="Value 1") fig.add_scatter(x=df["time"], y=df["value2"], row=2, col=1, name="Value 2") fig.update_yaxes(secondary_y=True, range=[0, 1], fixedrange=True, showgrid=False, visible=False, row=1, col=1) fig.update_layout(title="Shapes with Multiple Subplots") fig.show()
Question: I have a single column in a big datasheet which I want to change, by extracting substrings from the string in that column. I do this by using str.extract on that column like so:    Groups     Group (A)   Group (B)   Group (CA)   Group (CB)   Group (G)   Group (XP)    What I want to get is the following:    Groups     (A)   (B)   (CA)   (CB)   (G)   (XP)    I did try doing this with str.extract as mentioned above, because the datasheet is transformed into a Dataframe for more data transformation stuff beforehand. Usually this works just fine, but in this case it doesn't. The relevant code is rule = "(\(A\)|\(B\)|\(G\)|\(CA\)|\(CB\)|\(XP\))" df["Groups"] = df["Groups"].str.extract(rule, expand=True)  For some ungodly reason, it only extracts (A) and (B), not (G), nor any other characters. What am I doing wrong? Edit: The code around the whole application is super convoluted, badly maintained and overall not all that stable, so the issue may be somewhere else. But since this is the first and only case where this happened in any transformation, because it usually works fine for other cases, and since I managed to isolate it pretty well, the mistake should be within the issue described above.

Answer: This works fine in my hands, you have to use a raw string (r'...') to avoid the DeprecationWarning: rule = r'(\(A\)|\(B\)|\(G\)|\(CA\)|\(CB\)|\(XP\))' df['out'] = df['Groups'].str.extract(rule, expand=True)  Another, more generic, option could be to allow anything between the parentheses, except parentheses themselves: rule = r'(\([^()]+\))' df['out2'] = df['Groups'].str.extract(rule, expand=True)  Or, using a non-greedy quantifier: rule = r'(\(.+?\))' df['out3'] = df['Groups'].str.extract(rule, expand=True)  Output:  Groups out out2 out3 0 Group (A) (A) (A) (A) 1 Group (B) (B) (B) (B) 2 Group (CA) (CA) (CA) (CA) 3 Group (CB) (CB) (CB) (CB) 4 Group (G) (G) (G) (G) 5 Group (XP) (XP) (XP) (XP)  Regex demos:  original regex (\([^()]+\)) (\(.+?\))  If, really, the first approach doesn't work for you, you might have hidden characters that cause the regex to fail.
Question: I have a huge dataframe and want to split it into small files for better performance. Here is the example code to write. BUT I can not just read a small pieces from it without loading whole dataframe into memory. import pandas as pd import os # Create a sample DataFrame with daily frequency data = { "timestamp": pd.date_range(start="2023-01-01", periods=1000, freq="D"), "value": range(100) } df = pd.DataFrame(data) # Add a column for year (to use as a partition key) df["year"] = df["timestamp"].dt.year df["month"] = df["timestamp"].dt.month # Use the join method to expand the DataFrame (Cartesian product with a multiplier) multiplier = pd.DataFrame({"replica": range(100)}) # Create a multiplier DataFrame expanded_df = df.join(multiplier, how="cross") # Cartesian product using cross join # Define the output directory output_dir = "output_parquet" # Save the expanded DataFrame to Parquet with year-based partitioning expanded_df.to_parquet( output_dir, partition_cols=["year", "month"], # Specify the partition column )  Which is the best way to read from the dataset if I only need data from 2023-12-01 to 2024-01-31?

Answer: You can load your dataset selectively if your parquet output is properly partitioned. You can use libraries like PyArrow to let you filter the data at the file or partition level to make sure only the relevant data is loaded in to memory. Here's how you can do it using pyarrow.dataset: import pyarrow.dataset as ds dataset = ds.dataset("output_parquet", format="parquet", partitioning="hive") filtered_table = dataset.to_table(filter=(ds.field("year") == 2023) & (ds.field("month") == 12)) # Or any other desired condition filtered_df = filtered_table.to_pandas()  You may also use pyarrow.parquet.ParquetDataset to achieve the same goal, but it's less optimized and a bit outdated: import pyarrow.parquet as pq dataset = pq.ParquetDataset("output_parquet", filters=[("year", "=", 2023), ("month", "=", 12)]) table = dataset.read() df = table.to_pandas()
Question: I was experimenting with Python Structural Pattern Matching and wanted to write a match statement capable of matching repeated occurrences in a sequence. Suppose that I have the following list of tuples and was trying to match every pair in which the first element of the tuple is the same. from itertools import combinations objs = (('a', 'b'), ('c', 'd', 'e'), ('f',), ('a', 'g'), ('b', 'a')) pairs = list(combinations(objs, 2))  The goal was my code to match the pair: (('a', 'b'), ('a', 'g')) as both tuples starts with 'a'. My first attempt was: from itertools import combinations objs = (('a', 'b'), ('c', 'd', 'e'), ('f',), ('a', 'g'), ('b', 'a')) pairs = list(combinations(objs, 2)) for pair in pairs: match pair: # This raises Syntax Error! case ((x, *y), (x, *z)): print('Found!') case _: pass  However this raises: SyntaxError: multiple assignments to name 'x' in pattern. A possible work around I guess is to use guards in which one gives different names such as x1 and x2 and then matches only if x1 == x2, however this can quickly become quite messy if I want to enforce more than one equality. In my opinion using the same binding name is both elegant and practical with the obvious implication that the two should be the same. And so I ask, is there any reason why Structural Pattern Matching enforces the variable names to be different? Is there a nicer way to accomplish this that I'm missing?

Answer: From PEP 635 – Structural Pattern Matching: Motivation and Rationale, section Capture Patterns:  A name used for a capture pattern must not coincide with another capture pattern in the same pattern. This, again, is similar to parameters, which equally require each parameter name to be unique within the list of parameters. It differs, however, from iterable unpacking assignment, where the repeated use of a variable name as target is permissible (e.g., x, x = 1, 2). The rationale for not supporting (x, x) in patterns is its ambiguous reading: it could be seen as in iterable unpacking where only the second binding to x survives. But it could be equally seen as expressing a tuple with two equal elements (which comes with its own issues). Should the need arise, then it is still possible to introduce support for repeated use of names later on.
Question: Consider the following small example (based off of this gallery example]): import altair as alt from vega_datasets import data import polars as pl # add a column indicating the year associated with each date source = pl.from_pandas(data.stocks()).with_columns(year=pl.col.date.dt.year()) # an MSFT specific plot msft_plot = ( alt.Chart(source.filter(pl.col.symbol.eq("MSFT"))) .mark_line() .encode(x="date:T", y="price:Q", color="year:O") ) # the original plot: https://altair-viz.github.io/gallery/line_chart_with_points.html all_plot = ( alt.Chart(source) .mark_line() .encode(x="date:T", y="price:Q", color="symbol:N") ) msft_plot & all_plot  This produces the following output:  On the other hand, if I only plot all_plot:  How do I stop the legends being merged when I concatenate msft_plot & all_plot?

Answer: You can use (msft_plot & all_plot).resolve_scale(color='independent'):  More info about the resolve_ methods can be found in this section of the docs.
Question: I hope someone can help me understand the difference between the "single" and "table" methods in pandas.DataFrame.quantile?  Whether to compute quantiles per-column (‘single’) or over all columns (‘table’). https://pandas.pydata.org/docs/reference/api/pandas.DataFrame.quantile.html  For example, the following code yields the same results. import numpy as np import pandas as pd df = pd.DataFrame(np.array([[1, 1], [2, 10], [3, 100], [4, 100]]), columns=["a", "b"]) print(df.quantile(method="single", interpolation="nearest")) print(df.quantile(method="table", interpolation="nearest"))  a 3 b 100 Name: 0.5, dtype: int64 a 3 b 100 Name: 0.5, dtype: int64 

Answer: If you invert the order of the values in b the result will be different: df['b'] = df['b'].values[::-1] print(df.quantile(method='single', interpolation='nearest')) print(df.quantile(method='table', interpolation='nearest'))  Output: a 3 b 100 Name: 0.5, dtype: int64 a 3 b 10 Name: 0.5, dtype: int64  The exact difference in behavior is a bit tricky to understand. This happens in core/frame.py:  if method == "single": res = data._mgr.quantile(qs=q, interpolation=interpolation) elif method == "table": valid_interpolation = {"nearest", "lower", "higher"} if interpolation not in valid_interpolation: raise ValueError( f"Invalid interpolation: {interpolation}. " f"Interpolation must be in {valid_interpolation}" ) # handle degenerate case if len(data) == 0: if data.ndim == 2: dtype = find_common_type(list(self.dtypes)) else: dtype = self.dtype return self._constructor([], index=q, columns=data.columns, dtype=dtype) q_idx = np.quantile(np.arange(len(data)), q, method=interpolation) by = data.columns if len(by) > 1: keys = [data._get_label_or_level_values(x) for x in by] indexer = lexsort_indexer(keys) else: k = data._get_label_or_level_values(by[0]) indexer = nargsort(k) res = data._mgr.take(indexer[q_idx], verify=False) res.axes[1] = q result = self._constructor_from_mgr(res, axes=res.axes) return result.__finalize__(self, method="quantile")  In short, if you use method='single', this computes: np.percentile(a, 50, axis=0, method='nearest') # array([ 3, 100])  With method='table', this computes the quantile on the indexer (np.arange(len(df))). The result with be an integer (q_idx) between 0 and len(df). The, this sorts the rows with lexsort_indexer, giving priority to the first column(s), and finally takes the (q_idx)th row: data = df q = 0.5 interpolation = 'nearest' q_idx = np.quantile(np.arange(len(data)), q, method=interpolation) # 2 indexer = lexsort_indexer([data._get_label_or_level_values(x) for x in data.columns]) # array([0, 1, 2, 3]) df.iloc[indexer] # a b # 0 1 100 # 1 2 100 # 2 3 10 # this row will be picked # 3 4 1 data.iloc[indexer[q_idx]] # a 3 # b 10 # Name: 2, dtype: int64  This means that the result is dependent on the order of the columns: print(df.quantile(method='table', interpolation='nearest')) a 3 b 10 Name: 0.5, dtype: int64 # now let's give b priority over a print(df.iloc[:, ::-1].quantile(method='table', interpolation='nearest')) b 100 a 1 Name: 0.5, dtype: int64
Question: When printing a polars data frame, pl.Duration are printed in a "human format" by default. What function is used to do this conversion? Is it possible to use it? Trying "{}".format() returns something readable but not as good. import polars as pl data = {"end": ["2024/11/13 10:28:00", "2024/10/10 10:10:10", "2024/09/13 09:12:29", "2024/08/31 14:57:02", ], "start": ["2024/11/13 10:27:33", "2024/10/10 10:01:01", "2024/09/13 07:07:07", "2024/08/25 13:48:28", ] } df = pl.DataFrame(data) df = df.with_columns( pl.col("end").str.to_datetime(), pl.col("start").str.to_datetime(), ) df = df.with_columns( duration = pl.col("end") - pl.col("start"), ) df = df.with_columns( pl.col("duration").map_elements(lambda t: "{}".format(t), return_dtype=pl.String()).alias("duration_str") ) print(df)  shape: (4, 4) ┌─────────────────────┬─────────────────────┬──────────────┬─────────────────┐ │ end ┆ start ┆ duration ┆ duration_str │ │ --- ┆ --- ┆ --- ┆ --- │ │ datetime[μs] ┆ datetime[μs] ┆ duration[μs] ┆ str │ ╞═════════════════════╪═════════════════════╪══════════════╪═════════════════╡ │ 2024-11-13 10:28:00 ┆ 2024-11-13 10:27:33 ┆ 27s ┆ 0:00:27 │ │ 2024-10-10 10:10:10 ┆ 2024-10-10 10:01:01 ┆ 9m 9s ┆ 0:09:09 │ │ 2024-09-13 09:12:29 ┆ 2024-09-13 07:07:07 ┆ 2h 5m 22s ┆ 2:05:22 │ │ 2024-08-31 14:57:02 ┆ 2024-08-25 13:48:28 ┆ 6d 1h 8m 34s ┆ 6 days, 1:08:34 │ └─────────────────────┴─────────────────────┴──────────────┴─────────────────┘ 

Answer: Polars 1.14.0 added Duration type support to .dt.to_string() It can produce iso and polars formatted strings. pl.select( pl.duration(hours=1, minutes=2).dt.to_string() # format="iso" ).item()  # 'PT1H2M'  pl.select( pl.duration(hours=1, minutes=2).dt.to_string(format="polars") ).item()  # '1h 2m'
Question: I'm trying to find if there is a way to render LaTeX formulas in Shiny for Python or any low-hanging fruit workaround for that. Documentation doesn't have any LaTeX mentions, so looks like there's no dedicated functionality to support it. Also double-checked different variations of Latex in their playground. Tried this but didn't work: from shiny.express import input, render, ui @render.text def txt(): equation = r"$$\[3 \times 3+3-3 \]$$".strip() return equation 

Answer: You can import Katex. I got here via https://stackoverflow.com/a/65540803/5599595. Running in shinylive from shiny.express import ui from shiny import render with ui.tags.head(): # Link KaTeX CSS ui.tags.link( rel="stylesheet", href="https://cdn.jsdelivr.net/npm/katex@0.16.0/dist/katex.min.css" ), ui.tags.script(src="https://cdn.jsdelivr.net/npm/katex@0.16.0/dist/katex.min.js"), ui.tags.script(src="https://cdn.jsdelivr.net/npm/katex@0.16.0/dist/contrib/auto-render.min.js"), ui.tags.script(""" document.addEventListener('DOMContentLoaded', function() { renderMathInElement(document.body); }); """) with ui.card(): ui.p("Here's a quadratic formula: \\[x = \\frac{-b \\pm \\sqrt{b^2 - 4ac}}{2a}\\]") ui.p("And an inline equation: \\(E = mc^2\\)") ui.p("\\[3 \\times 3+3-3 \\]")  Here's an alternative which can render the array running in shiny live: from shiny.express import ui from shiny import render with ui.tags.head(): ui.tags.link( rel="stylesheet", href="https://cdn.jsdelivr.net/npm/katex@0.16.0/dist/katex.min.css" ), ui.tags.script(src="https://cdn.jsdelivr.net/npm/katex@0.16.0/dist/katex.min.js"), ui.tags.script(src="https://cdn.jsdelivr.net/npm/katex@0.16.0/dist/contrib/auto-render.min.js"), ui.tags.script(""" document.addEventListener('DOMContentLoaded', function() { renderMathInElement(document.body, { delimiters: [ {left: "$$", right: "$$", display: true}, {left: "\\[", right: "\\]", display: true}, {left: "$", right: "$", display: false}, {left: "\\(", right: "\\)", display: false} ] }); }); """) with ui.card(): ui.p("Here's a quadratic formula: $$x = \\frac{-b \\pm \\sqrt{b^2 - 4ac}}{2a}$$") ui.p("And an inline equation: $$E = mc^2$$") ui.p("And simple math $$3 \\times 3+3-3$$") # KaTeX tables https://www.redgregory.com/notion/2020/12/23/a-katex-table-cheatsheet-for-notion ui.p("Table 1 $$ \\begin{array}{cc} a & b \\\\ c & d \\end{array} $$") ui.p("""Table 2 $$ \\begin{array} {|c|c|} \\hline A & B \\\\ \\hline 1 & 2 \\\\ \\hline 3 & 4 \\\\ \\hline \\end{array} $$ """.strip())
Question: If I use such data import networkx as nx G = nx.Graph() G.add_nodes_from([1, 2, 3, 4, 5, 6, 7]) G.add_edges_from([(1, 2), (1, 3), (2, 4), (5, 6)]) print(list(nx.connected_components(G)))  Everything works fine. But what if I need to get connected values from multiple tuple, such as the folowing import networkx as nx G = nx.Graph() G.add_nodes_from([1, 2, 3, 4, 5, 6, 7]) G.add_edges_from([(1, 2), (1, 3, 7), (2, 4, 1, 6), (5, 6)]) print(list(nx.connected_components(G)))  As you can see its not classic and not working. What methods can I implement in order to pass such data, so that I got connected array values? I expect getting arrays with connected values between each other

Answer: The issue is that add_edges_from only takes a list of (source, target) tuples. You could use itertools: import networkx as nx from itertools import chain, pairwise G = nx.Graph() G.add_nodes_from([1, 2, 3, 4, 5, 6, 7]) edges = [(1, 2), (1, 3, 7), (2, 4, 1, 6), (5, 6)] G.add_edges_from(chain.from_iterable(map(pairwise, edges))) print(list(nx.connected_components(G)))  Variant without import: G.add_edges_from(e for t in edges for e in zip(t, t[1:]))  Output: [{1, 2, 3, 4, 5, 6, 7}]  Graph:  Intermediates: # list(chain.from_iterable(map(pairwise, edges))) # [e for t in edges for e in zip(t, t[1:])] [(1, 2), (1, 3), (3, 7), (2, 4), (4, 1), (1, 6), (5, 6)]