eshangj/stackoverflow_q_and_a_sample huggingface dataset:
Question: As an example: import pandas as pd df = pd.DataFrame({ "Hello World": [1, 2, 3, 4], "And Some More": [10.0, 20.0, 30.0, 40.0], }) df_caption = "Table 1: My Table" df.style.set_caption(df_caption) # only works for HTML; https://stackoverflow.com/q/57958432 with pd.option_context('display.max_rows', None, 'display.max_columns', None, 'display.width', None, 'max_colwidth', 50, 'display.float_format', "{:.2f}".format): df_str = df.to_string() print(df_str)  ... outputs:  Hello World And Some More 0 1 10.00 1 2 20.00 2 3 30.00 3 4 40.00  ... and clearly, there is no table title/caption in the plain text output of .to_string(). Sure I can just print(df_caption) myself separately - but is it otherwise somehow possible to add dataframe (table) caption on the Pandas DataFrame object, so that it is output in the string generated by .to_string()?

Answer:  DataFrame.style has a specific use that does not relate to printing DataFrames in the console. From the code documentation:   Contains methods for building a styled HTML representation of the DataFrame.   DataFrame.to_string() has many attributes, but none of them relate to displaying a caption or a name. It does take a header, but that relates specifically to the column names.  DataFrame.__repr__ uses DataFrame.to_string, so no captions here either.   In conclusion: it's not "possible to add dataframe (table) caption on the Pandas DataFrame object, so that it is output in the string generated by .to_string()".  You can, of course, create your own function to do that: data = { "Name": ["Alice", "Bob", "Charlie", "David", "Emily"], "Age": [25, 30, 35, 40, 45], "City": ["New York", "Los Angeles", "Chicago", "Houston", "Boston"], } df = pd.DataFrame(data) def print_df(df, name): print(df) print(f"{name = }") print_df(df, name="Example DataFrame")   Name Age City 0 Alice 25 New York 1 Bob 30 Los Angeles 2 Charlie 35 Chicago 3 David 40 Houston 4 Emily 45 Boston name = 'Example DataFrame'
Question: Ned help to parse a csv file which has following details to xml file csv file is in the following format: name,ip,timeout domain\user1,10.119.77.218,9000 domain\user2,2.80.189.26,9001 domain\user3,4.155.10.110,9002 domain\user4,9.214.119.86,9003 domain\user5,4.178.187.27,9004 domain\user6,3.76.178.117,9005  The above details from csv needs to be added to XML file which has the following format: <login> <entry name="domain\user1" ip="10.119.77.218" timeout="9000"/> <entry name="domain\user2" ip="2.80.189.26" timeout="9001"/> <entry name="domain\user11" ip="4.155.10.110" timeout="12000"/> ... </login>  Need some script because there are tones of files which needs to be converted. I tried the following tool: https://www.convertcsv.com/csv-to-xml.htm But the above tool converts individual row to separate entry which is not needed. Looking for your feedback. Thank you.

Answer: If the structure of CSV/XML file is simple you can use csv module and construct the string directly (for more complicated scenarios I recommend lxml/bs4 modules): import csv with open("your_data.csv", "r") as f: reader = csv.reader(f) header = next(reader) data = list(reader) out = ["<login>"] for line in data: s = " ".join(f'{k}="{v}"' for k, v in zip(header, line)) out.append(f"\t<entry {s} />") out.append("</login>") print(*out, sep="\n")  Prints: <login> <entry name="domain\user1" ip="10.119.77.218" timeout="9000" /> <entry name="domain\user2" ip="2.80.189.26" timeout="9001" /> <entry name="domain\user3" ip="4.155.10.110" timeout="9002" /> <entry name="domain\user4" ip="9.214.119.86" timeout="9003" /> <entry name="domain\user5" ip="4.178.187.27" timeout="9004" /> <entry name="domain\user6" ip="3.76.178.117" timeout="9005" /> </login>
Question: How can I aggreate all product names of the grouped dataframe into a new column as list or set: import pandas as pd # 2.0.3 df = pd.DataFrame( { "customer_id": [1, 2, 3, 2, 1], "order_id": [1, 2, 3, 4, 1], "products": ["foo", "bar", "baz", "foo", "bar"], "amount": [1, 1, 1, 1, 1] } ) print(df) grouped = df.groupby(["customer_id", "order_id"]) df["product_order_count"] = grouped["amount"].transform("sum") df["all_products"] = grouped["products"].agg(list).reset_index() print(df)  Although I followed another question (Pandas groupby: How to get a union of strings) an exception is thrown: Traceback (most recent call last): File "C:\temp\tt.py", line 15, in <module> df["all_orders"] = grouped["products"].agg(list).reset_index() File "c:\Users\foo\.venvs\kapa_monitor-38\lib\site-packages\pandas\core\frame.py", line 3940, in __setitem__ self._set_item_frame_value(key, value) File "c:\Users\foo\.venvs\kapa_monitor-38\lib\site-packages\pandas\core\frame.py", line 4094, in _set_item_frame_value raise ValueError( ValueError: Cannot set a DataFrame with multiple columns to the single column all_products  Expected output (all_products, as list or set):  customer_id order_id products amount product_order_count all_products 0 1 1 foo 1 2 'foo', 'bar' 1 2 2 bar 1 1 'bar' 2 3 3 baz 1 1 'baz' 3 2 4 foo 1 1 'foo' 4 1 1 bar 1 2 'foo', 'bar' 

Answer: You could use transform with a function that returns something that is the same length with the group: df["all_products"] = grouped["products"].transform(lambda x: [list(x)]*len(x))  Output:  customer_id order_id products amount product_order_count all_products 0 1 1 foo 1 2 [foo, bar] 1 2 2 bar 1 1 [bar] 2 3 3 baz 1 1 [baz] 3 2 4 foo 1 1 [foo] 4 1 1 bar 1 2 [foo, bar]  Or you can joint the strings (I don't really recommend lists in the data): df["all_products"] = grouped["products"].transform(','.join)  which gives  customer_id order_id products amount product_order_count all_products 0 1 1 foo 1 2 foo,bar 1 2 2 bar 1 1 bar 2 3 3 baz 1 1 baz 3 2 4 foo 1 1 foo 4 1 1 bar 1 2 foo,bar
Question: I have two columns: serial_number and inv_number containing lists. If there is one inv_number for multiple serial_number, I need to make the size of inv_number's list the same as serial_number's.  serial_number inv_number 28 [С029768, С029775] [101040031171, 101040031172] 29 [090020960190402011, 090020960190402009] [210134002523, 210134002524] 31 [1094] [410124000215] 32 [01] [101040022094] 33 [F161B5, F17D86, F17D8D, F1825C, F1825A, F1825D] [101040026976]  Here at the index 33 we have 6 serial numbers but one inventory number, so it should be changed to [101040026976, 101040026976, 101040026976, 101040026976, 101040026976, 101040026976]  I've tried to do it by "multiplying" values to make a list (like [value] * N): si.loc[si['inv_number'].apply(len)==1, 'inv_number'].apply (lambda x: [str(x[0])] * si['serial_number'].apply(len).values)  but it gives me an error:  UFuncTypeError: ufunc 'multiply' did not contain a loop with signature matching types (dtype('<U12'), dtype('int64')) -> None  How can I solve this problem?

Answer: Try: mask = (df["serial_number"].str.len() > 1) & (df["inv_number"].str.len() == 1) df.loc[mask, "inv_number"] = df["serial_number"].str.len() * df.loc[mask, "inv_number"] print(df)  Prints:  serial_number inv_number 28 [С029768, С029775] [101040031171, 101040031172] 29 [090020960190402011, 090020960190402009] [210134002523, 210134002524] 31 [1094] [410124000215] 32 [01] [101040022094] 33 [F161B5, F17D86, F17D8D, F1825C, F1825A, F1825D] [101040026976, 101040026976, 101040026976, 101040026976, 101040026976, 101040026976]
Question: Consider this example: import pandas as pd df = pd.DataFrame({ "LIDSA": [0, 1, 2, 3], "CAE": [3, 5, 7, 9], "FILA": [1, 2, 3, 4], # 2 is default, so table idx 1 is default "VUAMA": [0.5, 1.0, 1.5, 2.0], }) df_colnames = { # https://stackoverflow.com/q/48243818 "LIDSA": "Lorem ipsum dolor sit amet", "CAE": "Consectetur adipiscing elit", "FILA": "Fusce imperdiet libero arcu", "VUAMA": "Vitae ultricies augue molestie ac", } # "Pandas autodetects the size of your terminal window if you set pd.options.display.width = 0" https://stackoverflow.com/q/11707586 with pd.option_context('display.max_rows', None, 'display.max_columns', None, 'display.width', 0, 'max_colwidth', 20, 'display.float_format', "{:.2f}".format): df_str = df.rename(df_colnames,axis=1).to_string() print(df_str)  This results with the terminal stdout printout, at the time 111 characters wide:  Lorem ipsum dolor sit amet Consectetur adipiscing elit Fusce imperdiet libero arcu Vitae ultricies augue molestie ac 0 0 3 1 0.50 1 1 5 2 1.00 2 2 7 3 1.50 3 3 9 4 2.00  So, only the last column got line-broken (and correspondingly, the values for it). I would have preferred that each long column name gets line-broken / word-wrapped at say 20 characters, and then the values output correspondingly, something like:  Lorem ipsum dolor Consectetur Fusce imperdiet Vitae ultricies sit amet adipiscing elit libero arcu augue molestie ac 0 0 3 1 0.50 1 1 5 2 1.00 2 2 7 3 1.50 3 3 9 4 2.00  I thought 'max_colwidth', 20 would do that, but apparently it doesn't. I even tried adding explicit linebreaks in the long column names, but they just get rendered as \n, and the column name is still in one line (as noted also in Linebreaks in pandas column names) So, is it possible to "word-wrap"/"line break" long column names in Pandas for plain text string output?

Answer: You could use textwrap.wrap and tabulate: # pip install tabulate from textwrap import wrap from tabulate import tabulate df_colnames_wrap = {k: '\n'.join(wrap(v, 20)) for k,v in df_colnames.items()} print(tabulate(df.rename(columns=df_colnames_wrap), headers='keys', tablefmt='plain'))  Output:  Lorem ipsum dolor Consectetur Fusce imperdiet Vitae ultricies sit amet adipiscing elit libero arcu augue molestie ac 0 0 3 1 0.5 1 1 5 2 1 2 2 7 3 1.5 3 3 9 4 2  With float formatting: print(tabulate(df.rename(columns=df_colnames_wrap) .convert_dtypes(), headers='keys', tablefmt='plain', floatfmt='.2f' ))  Output:  Lorem ipsum dolor Consectetur Fusce imperdiet Vitae ultricies sit amet adipiscing elit libero arcu augue molestie ac 0 0 3 1 0.50 1 1 5 2 1.00 2 2 7 3 1.50 3 3 9 4 2.00
Question:  I'd like to add new empty columns (elements of mylist) to an existing dataframe. This code does it: import polars as pl df = pl.DataFrame({'a': [1,2,3]}) mylist = [f'col{i}' for i in range(1,4)] data = [[''] for i in range(1,len(mylist)+1)] df.join(pl.DataFrame(data=data,schema=mylist), how='cross')  But is there a more polars way to do this? Something using .with_columns() or pl.any()? So that I do not have to create a new dataframe and join it?

Answer: You seem to be looking for pl.lit() which allows you to create an "expression" from a literal value. You can then use .alias() to choose the name. df.with_columns(pl.lit('').alias(col) for col in mylist)  shape: (3, 4) ┌─────┬──────┬──────┬──────┐ │ a ┆ col1 ┆ col2 ┆ col3 │ │ --- ┆ --- ┆ --- ┆ --- │ │ i64 ┆ str ┆ str ┆ str │ ╞═════╪══════╪══════╪══════╡ │ 1 ┆ ┆ ┆ │ │ 2 ┆ ┆ ┆ │ │ 3 ┆ ┆ ┆ │ └─────┴──────┴──────┴──────┘  map() could also be used instead of a comprehension. df.with_columns(map(pl.lit('').alias, mylist))
Question: When you try to convert a Torch bfloat16 tensor to a numpy array, it throws a TypeError: import torch x = torch.Tensor([0]).to(torch.bfloat16) x.numpy() # TypeError: Got unsupported ScalarType BFloat16 import numpy as np np.array(x) # same error  Is there a work-around to make this conversion?

Answer: Currently, numpy does not support bfloat16**. One work-around is to upcast the tensor from half-precision to single-precision before making the conversion: x.float().numpy()  The Pytorch maintainers are also considering adding a force=True option to the Tensor.numpy method to this automatically. ** although that may change thanks to work by @jakevdp
Question: I am trying to cross merge two dataframes but limiting the merge so only combinations within the same group are provided. The pandas documentation says When performing a cross merge, no column specifications to merge on are allowed. At the moment to achieve this I'm using a for loop and concatenating the resulting dfs but is there a more efficient way? Example input data: import pandas as pd df1 = pd.DataFrame({ 'group': [1, 1, 2, 2], 'field_a': ['apple', 'pear', 'banana', 'papaya'] }) df2 = pd.DataFrame({ 'group': [1, 1, 2, 2], 'field_b': ['apple', 'strawberry', 'coconut', 'papaya'] })  Example required output: pd.DataFrame({'group': [1, 1, 1, 1, 2, 2, 2, 2], 'field_a': ['apple', 'apple', 'pear', 'pear', 'banana', 'banana', 'papaya', 'papaya'], 'field_b': ['apple', 'strawberry', 'apple', 'strawberry', 'coconut', 'papaya', 'coconut', 'papaya']})  Current approach: cols = ['group', 'field_a', 'field_b'] all_possible_matches = pd.DataFrame({ col: [] for col in cols }) for group in [1, 2]: combined = df1[df1['group'] == group].merge(df2[df2['group'] == group][['field_b']], how='cross') all_possible_matches = pd.concat([all_possible_matches, combined]) 

Answer: A cross-merge by group would be equivalent to a merge on the group: out = df1.merge(df2, on='group') # if "group" is the only common column # out = df1.merge(df2)  Output:  group field_a field_b 0 1 apple apple 1 1 apple strawberry 2 1 pear apple 3 1 pear strawberry 4 2 banana coconut 5 2 banana papaya 6 2 papaya coconut 7 2 papaya papaya  Before how='cross' was available in pandas, a way to perform a cross-merge was actually to add a dummy key and merge on that: # before df1['key'] = 1 df2['key'] = 1 df1.merge(df2, on='key').drop(columns=['key']) # now df1.merge(df2, how='cross')
Question: Following the documentation for reading from cloud storage, I have created the below script that fails. import boto3 import polars as pl import os session = boto3.Session(profile_name=os.environ["AWS_PROFILE"]) credentials = session.get_credentials() current_credentials = credentials.get_frozen_credentials() # Specify your S3 bucket and file path s3_bucket = "bucket" s3_file_path = "path/file.parquet" # Create the full S3 path s3_path = f"s3://{s3_bucket}/{s3_file_path}" storage_options = { 'aws_access_key_id': current_credentials.access_key, 'aws_secret_access_key': current_credentials.secret_key, 'aws_region': 'us-east-1' credentials } df = pl.scan_parquet(s3_path, storage_options=storage_options)  This gives output below which I understand is a common error for not having permissions to access the file.  ComputeError: Generic S3 error: Client error with status 403 Forbidden: No Body  versions:  Python '3.9.18' polars '0.20.14' boto3 '1.34.58'  Running on macos. I am also successfully able to read the parquet using pandas just setting the AWS_PROFILE env var. Am I using the storage_options incorrectly? It doesn't seem able to take 'aws_profile' key value pair to extract local config credentials its self?

Answer: You might also need to pass the corresponding session token as follows. import polars as pl import boto3 profile_name = "your-profile" s3_path = "s3://my-s3-bucket/my_file.parquet" session = boto3.session.Session(profile_name=profile_name) credentials = session.get_credentials().get_frozen_credentials() df = pl.read_parquet( s3_path, storage_options={ "aws_access_key_id": credentials.access_key, "aws_secret_access_key": credentials.secret_key, "aws_session_token": credentials.token, "aws_region": session.region_name, }, )  From the AWS access keys documentation:  Specifies an AWS session token used as part of the credentials to authenticate the user. You receive this value as part of the temporary credentials returned by successful requests to assume a role. A session token is required only if you manually specify temporary security credentials. However, we recommend you always use temporary security credentials instead of long-term credentials. For security recommendations, see Security best practices in IAM.
Question: This is my DataFrame: import pandas as pd df = pd.DataFrame({'a': [150, 106, 119, 131, 121, 140, 160, 119, 170]})  And this is the expected output. I want to create column b:  a b 0 150 140 1 106 160 2 119 160 3 131 161 4 121 NaN 5 140 NaN 6 160 NaN 7 119 NaN 8 170 NaN  I want to get the maximum value in a rolling window of 6. However I want to ignore the first value of each window. In this picture I have shown the windows that I want. Red cells are the ones that should be excluded from calculations and green ones are the maximum value of the window that are in b.  I prefer a generic solution. For example getting max() after ignoring first N values of each window. These are some of my attempts that did not work: # attempt 1 df['b'] = df.a.shift(-1).rolling(6).max() # attempt 2 df['b'] = df.a.rolling(6, closed='left').max() # attempt 3 for i in range(3): x = df.iloc[i+1:i+6] 

Answer: The naive approach would be to use groupby.transform and drop the first item: N = 6 df['out'] = df.loc[::-1, 'a'].rolling(N).apply(lambda x: x.iloc[:-1].max())  However, since your operation is independent of this value, better compute the rolling.max on N-1 and shift after the fact. Which can simplify the code to: N = 6 df['out'] = df.loc[::-1, 'a'].rolling(N-1).max().shift()  Note that since rolling uses the previous values by default, we need to first invert the order of the Series with [::-1]. Output:  a out 0 150 140.0 1 106 160.0 2 119 160.0 3 131 170.0 4 121 NaN 5 140 NaN 6 160 NaN 7 119 NaN 8 170 NaN  generalization To generalize to skipping skip values: N = 6 skip = 2 df['out'] = df.loc[::-1, 'a'].rolling(N-skip).max().shift(skip)  Example (changing the second 119 to 118 for clarity):  a skip=0 skip=1 skip=2 skip=3 skip=4 skip=5 skip=6 0 150 150.0 140.0 140.0 140.0 140.0 140.0 NaN 1 106 160.0 160.0 160.0 160.0 160.0 160.0 NaN 2 119 160.0 160.0 160.0 160.0 160.0 118.0 NaN 3 131 170.0 170.0 170.0 170.0 170.0 170.0 NaN 4 121 NaN NaN NaN NaN NaN NaN NaN 5 140 NaN NaN NaN NaN NaN NaN NaN 6 160 NaN NaN NaN NaN NaN NaN NaN 7 118 NaN NaN NaN NaN NaN NaN NaN 8 170 NaN NaN NaN NaN NaN NaN NaN
Question: I am trying to do a series of operations on a single column in a lazy polars DataFrame and I am trying to avoid using with_columns_seq repeatedly in doing so, but there's a ComputeError stating a duplicate column name. Is there a better alternative to this? df = ( df .with_columns_seq([ pl.col('sentiment').cast(pl.UInt8), pl.col('review').map_elements(lambda x: BeautifulSoup(x).get_text()), pl.col('review').str.replace(r"[^a-zA-Z0-9]", " "), pl.col('review').str.to_lowercase(), # pl.col('review').str.split_by(' ') ]) ) df.collect().head()  and the error is ComputeError: the name: 'review' passed to `LazyFrame.with_columns` is duplicate It's possible that multiple expressions are returning the same default column name. If this is the case, try renaming the columns with `.alias("new_name")` to avoid duplicate column names.  tried to compose all operations but failed due to a duplicate column error

Answer: Explanation Your error message tells you already almost everything to grasp the problem:  It's possible that multiple expressions are returning the same default column name. If this is the case, try renaming the columns with .alias("new_name") to avoid duplicate column names.  However, it does not in this case actually suggest a solution that you probably want. The error comes because each of the lines starting with pl.col("review") creates a new column review, because .with_columns_seq does not mean that you run an each of those expressions to updated data one after another. It just means that parallel processing is not used to do this operation, and you might as well just use with_columns. The usual solution would be to use alias("new_name"), like the error message helpfully suggests. However, in your situation it would just create two columns with slightly differing data  .with_columns_seq([ pl.col('review').map_elements(lambda x: x), pl.col('review').str.to_lowercase().alias("lowercase"), ])  ┌───────────┬──────────────────────┬──────────────────────┐ │ sentiment ┆ review ┆ lowercase │ │ --- ┆ --- ┆ --- │ │ i64 ┆ str ┆ str │ ╞═══════════╪══════════════════════╪══════════════════════╡ │ 1 ┆ I love this movie ┆ i love this movie │ │ 0 ┆ I hate this movie!! ┆ i hate this movie!! │ ...  This seems to be not something you want. Instead you need a way to do each of those operations to the new review column sequantially. Solution To actually run these expressions for review sequentially, you pipe them together.  The power of expressions is that every expression produces a new expression, and that they can be piped together. You can run an expression by passing them to one of Polars execution contexts.  So you would do this instead: df = ( df .with_columns([ pl.col('sentiment').cast(pl.UInt8), pl.col('review').map_elements(lambda x: BeautifulSoup(x).get_text()) .str.replace_all(r"[^a-zA-Z0-9]", " ") .str.to_lowercase() ]) )
Question:  I'd like to parse lines of text to multiple columns and lines in polars, with user defined function. import polars as pl df = pl.DataFrame({'file': ['aaa.txt','bbb.txt'], 'text': ['my little pony, your big pony','apple+banana, cake+coke']}) def myfunc(p_str: str) -> list: res = [] for line in p_str.split(','): x = line.strip().split(' ') res.append({f'word{e+1}': w for e, w in enumerate(x)}) return res  If I just run a test it's fine, list of dicts is created: myfunc(df['text'][0]) [{'word1': 'my', 'word2': 'little', 'word3': 'pony'}, {'word1': 'your', 'word2': 'big', 'word3': 'pony'}]  Even creating a dataframe of it is easy: pl.DataFrame(myfunc(df['text'][0]))  But trying to do map_elements() fails: (df.with_columns(pl.struct(['text']).map_elements(lambda x: myfunc(x['text'])).alias('aaa') ) )  thread '' panicked at crates/polars-core/src/chunked_array/builder/list/anonymous.rs:161:69: called Result::unwrap() on an Err value: InvalidOperation(ErrString("It is not possible to concatenate arrays of different data types.")) --- PyO3 is resuming a panic after fetching a PanicException from Python. --- What I'd like as a result is something like: file word1 word2 word3 aaa.txt my little pony aaa.txt your big pony bbb.txt apple+banana bbb.txt cake+coke  Any idea?

Answer: The error is likely due to the complex return type returned by the UDF. Polars quickly infers the dtype of the inner dictionaries. However, the dictionaries being evaluated later might have a different number of fields and the error is raised. A simpler reproducible example would be the following. import random import polars as pl def ufunc(x: int): return [ {f"word_{i}": "elephant" for i in range(random.randint(1, 4))} for _ in range(random.randint(1, 4)) ] pl.DataFrame({"id": [1, 2]}).with_columns(pl.col("id").map_elements(ufunc))  Originally, I thought correctly setting the returned_dtype parameter of pl.Expr.map_elements would resolve the issue. However, I didn't find a way to specify a struct dtype with varying number of fields. Still, when wrapping the return value into another list, polars seems to infer the returned dtype correctly. Then, we can obtain the first (and only) list element using .list.first, explode the lines into rows using pl.DataFrame.explode, and finally unnest the inner structs. ( pl.DataFrame({"id": [1, 2]}) .with_columns( pl.col("id").map_elements(lambda x: [ufunc(x)]).list.first() ) .explode("id") .unnest("id") )  shape: (7, 4) ┌──────────┬──────────┬──────────┬──────────┐ │ word_0 ┆ word_1 ┆ word_2 ┆ word_3 │ │ --- ┆ --- ┆ --- ┆ --- │ │ str ┆ str ┆ str ┆ str │ ╞══════════╪══════════╪══════════╪══════════╡ │ elephant ┆ elephant ┆ null ┆ null │ │ elephant ┆ elephant ┆ elephant ┆ elephant │ │ elephant ┆ null ┆ null ┆ null │ │ elephant ┆ null ┆ null ┆ null │ │ elephant ┆ elephant ┆ null ┆ null │ │ elephant ┆ elephant ┆ elephant ┆ null │ │ elephant ┆ null ┆ null ┆ null │ └──────────┴──────────┴──────────┴──────────┘