eshangj/stackoverflow_q_and_a_sample huggingface dataset:
Question: I'm looking for an efficient way to copy / replace parts of a dataframe with other parts of the same dataframe in Polars. For instance, in the following minimal example dataframe pl.DataFrame({ "year": [2020,2021,2020,2021], "district_id": [1,2,1,2], "distribution_id": [1, 1, 2, 2], "var_1": [1,2,0.1,0.3], "var_N": [1,2,0.3,0.5], "unrelated_var": [0.2,0.5,0.3,0.7], })  I'd like to replace all column values of "var_1" & "var_N" where the "distribution_id" = 2 with the corresponding values where the "distribution_id" = 1. This is the desired result: pl.DataFrame({ "year": [2020,2021,2020,2021], "district_id": [1,2,1,2], "distribution_id": [1, 1, 2, 2], "var_1": [1,2,1,2], "var_N": [1,2,1,2], "unrelated_var": [0.2,0.5,0.3,0.7], })  I tried to use a "when" expression, but it fails with "polars.exceptions.ShapeError: shapes of self, mask and other are not suitable for zip_with operation" df = df.with_columns([ pl.when(pl.col("distribution_id") == 2).then(df.filter(pl.col("distribution_id") == 1).otherwise(pl.col(col)).alias(col) for col in columns_to_copy ] )  Here's what I used to do with SQLAlchemy: table_alias = table.alias("table_alias") stmt = table.update().\ where(table.c.year == table_alias.c.year).\ where(table.c.d_id == table_alias.c.d_id).\ where(table_alias.c.distribution_id == 1).\ where(table.c.distribution_id == 2).\ values(var_1=table_alias.c.var_1, var_n=table_alias.c.var_n)  Thanks a lot for you help!

Answer: You could filter the 1 columns, change their id to 2 and discard the unneeded columns. df.filter(distribution_id = 1).select( "year", "district_id", "^var_.+$", distribution_id = pl.lit(2, pl.Int64) )  shape: (2, 5) ┌──────┬─────────────┬───────┬───────┬─────────────────┐ │ year ┆ district_id ┆ var_1 ┆ var_N ┆ distribution_id │ │ --- ┆ --- ┆ --- ┆ --- ┆ --- │ │ i64 ┆ i64 ┆ f64 ┆ f64 ┆ i64 │ ╞══════╪═════════════╪═══════╪═══════╪═════════════════╡ │ 2020 ┆ 1 ┆ 1.0 ┆ 1.0 ┆ 2 │ │ 2021 ┆ 2 ┆ 2.0 ┆ 2.0 ┆ 2 │ └──────┴─────────────┴───────┴───────┴─────────────────┘   (note: "^var_.+$" selects columns by regex, but selectors can be used if preferred.)  With the data "aligned", you can pass it to .update() df.update( df.filter(distribution_id = 1) .select("year", "district_id", "^var_.+$", distribution_id = pl.lit(2, pl.Int64)), on=["year", "district_id", "distribution_id"] )  shape: (4, 6) ┌──────┬─────────────┬─────────────────┬───────┬───────┬───────────────┐ │ year ┆ district_id ┆ distribution_id ┆ var_1 ┆ var_N ┆ unrelated_var │ │ --- ┆ --- ┆ --- ┆ --- ┆ --- ┆ --- │ │ i64 ┆ i64 ┆ i64 ┆ f64 ┆ f64 ┆ f64 │ ╞══════╪═════════════╪═════════════════╪═══════╪═══════╪═══════════════╡ │ 2020 ┆ 1 ┆ 1 ┆ 1.0 ┆ 1.0 ┆ 0.2 │ │ 2021 ┆ 2 ┆ 1 ┆ 2.0 ┆ 2.0 ┆ 0.5 │ │ 2020 ┆ 1 ┆ 2 ┆ 1.0 ┆ 1.0 ┆ 0.3 │ │ 2021 ┆ 2 ┆ 2 ┆ 2.0 ┆ 2.0 ┆ 0.7 │ └──────┴─────────────┴─────────────────┴───────┴───────┴───────────────┘
Question: I have data where for multiple years, observations i are categorized in cat. An observation i can be in multiple categories in any year, but is unique across years. I am trying to count unique values for i by year, by cat, and by year and cat. I'm learning Python (v3.12) & Pandas (v2.2.1). I can make this work, but only by creating separate tables for the counts, and merging them back in with the main data. See the example below. I suspect there is a better way to do this. Is there, and, if so, how? import pandas as pd df = pd.DataFrame( {'year': [2020,2020,2020,2021,2021,2022,2023,2023,2023,2023], 'cat': [1,1,2,2,3,3,1,2,3,4], 'i': ['a','a','b','c','d','e','f','f','g','g'] }) df df_cat = df.groupby('cat')['i'].nunique() df_year = df.groupby('year')['i'].nunique() df_catyear = df.groupby(['cat', 'year'])['i'].nunique() df_merged = df.merge(df_cat, how='left', on='cat').rename(columns={'i_x': 'i', 'i_y': 'n_by_cat'}) df_merged = df_merged.merge(df_year, how='left', on='year').rename(columns={'i_x': 'i', 'i_y': 'n_by_year'}) df_merged = df_merged.merge(df_catyear, how='left', on=['cat', 'year']).rename(columns={'i_x': 'i', 'i_y': 'n_by_catyear'}) 

Answer: You could use a simple loop and groupby.transform: groups = ['cat', 'year', ['cat', 'year']] for g in groups: df[f"n_by_{''.join(g)}"] = df.groupby(g)['i'].transform('nunique')  Output:  year cat i n_by_cat n_by_year n_by_catyear 0 2020 1 a 2 2 1 1 2020 1 a 2 2 1 2 2020 2 b 3 2 1 3 2021 2 c 3 2 1 4 2021 3 d 3 2 1 5 2022 3 e 3 1 1 6 2023 1 f 2 2 1 7 2023 2 f 3 2 1 8 2023 3 g 3 2 1 9 2023 4 g 1 2 1
Question: I wrote a docker compose file and used docker compose up -d command. Then I wrote a simple python script to connect with maria db but I get error everytime. mariadb version in my virtual environment is 1.0.11 pip install mariadb==1.0.11 version: '3.8' services: mariadb: image: mariadb:latest container_name: my_mariadb restart: always environment: MYSQL_ROOT_PASSWORD: myrpass MYSQL_DATABASE: db1 MYSQL_USER: user MYSQL_PASSWORD: mypass volumes: - mariadb-data:/var/lib/mysql ports: - "3307:3307" volumes: mariadb-data: {}  My python code: import mariadb host = "mariadb" port = 3307 user = "user" password = "mypass" database = "db1" try: conn = mariadb.connect( host=host, port=port, user=user, password=password, database=database ) cursor = conn.cursor() cursor.execute("SELECT VERSION()") version = cursor.fetchone()[0] print(f"Connected to MariaDB server version: {version}") except mariadb.Error as e: print(f"Error connecting to database: {e}") else: print('Connection found attempting to close it now') if conn: conn.cursor().close() conn.close() finally: print('Code execution complete')  Error when I use mariadb as host: Error connecting to database: Unknown MySQL server host 'mariadb' (-3) Code execution complete  Error when I use localhost as host: Error connecting to database: Access denied for user 'user'@'localhost' (using password: YES) Code execution complete 

Answer:  version: '3.8' services: mysql: image: mysql:latest container_name: db_mysql restart: always environment: MYSQL_ROOT_PASSWORD: r_pass MYSQL_AUTHENTICATION_PLUGIN: 'mysql_native_password' # ... other environment variables volumes: - mysql-data:/var/lib/mysql ports: - "3306:3306" mariadb: image: mariadb:latest container_name: db_mariadb restart: always environment: MYSQL_ROOT_PASSWORD: r_pass MYSQL_AUTHENTICATION_PLUGIN: 'mysql_native_password' # ... other environment variables volumes: - mariadb-data:/var/lib/mysql ports: - "3307:3306" volumes: mysql-data: {} mariadb-data: {}  Here is my docker-compose.yml file Create docker compose images docker compose up -d  Stop docker compose docker compose down -v  Enter inside docker containers for mysql docker exec -it db_mysql mysql -u root -p  mariadb docker exec -it db_mariadb mariadb -u root -p  password for mysql and mariadb r_pass  Prerequisites if you are a ubuntu user sudo apt-get install libmariadb-dev pip3 install mariadb==1.0.11  Python code # Module Imports import mariadb import sys # Connect to MariaDB Platform try: conn = mariadb.connect( user="root", password="r_pass", host="<docker_ip>", # docker inspect db_mariadb port=3306, database="test" ) except mariadb.Error as e: print(f"Error connecting to MariaDB Platform: {e}") sys.exit(1) # Get Cursor cur = conn.cursor()
Question: I'm reading some csv files where the column headers are pretty annoying: they contain whitespaces, tabs, etc. A B C D E CD E 300 0 0 0 CD E 1071 0 0 0 K E 390 0 0 0  I want to read the file, then remove all whitespaces and/or tabs from the column names. Currently I do import polars as pl file_df = pl.read_csv(csv_file, comment_prefix='#', separator='\t') file_df = file_df.rename(lambda column_name: column_name.strip())  Is this the "polaric" way to do it? I'm not a big fan of lambdas, but if the only other solution is to write a function just for this, I guess I'll stick to lambdas.

Answer: The solution is to use a function as you have shown. However, in the case of .strip() without arguments it can be simplified slightly. Another way to write the strip is by using str.strip() >>> " A ".strip() # 'A' >>> str.strip(" A ") # 'A'  str.strip and the lambda in this case do the same thing: one = lambda column: column.strip() two = str.strip  >>> one(" A ") # 'A' >>> two(" A ") # 'A'  df.rename() runs a function at the Python level, meaning we can pass str.strip directly. import polars as pl csv = b""" A \t B \t C 1\t2\t3 4\t5\t6 """ df = pl.read_csv(csv, separator="\t")  >>> df.columns # [' A ', ' B ', ' C'] >>> df.rename(str.strip).columns # ['A', 'B', 'C'] >>> df.rename(str.lower).columns # [' a ', ' b ', ' c']  It's only useful if you're calling functions without additional arguments. For anything more complex, you'll need to use a lambda (or def).
Question: I need the column names from the dataframe that contain both the term software and packages. I'm able to filter out columns containing one string.. for eg: software_cols = df.filter(regex='Software|software|SOFTWARE').columns How do I achieve the same by mentioning 'Packages/packages/PACKAGES' as well. Eligible column names should be like 'Local Software Packages', 'Software XYZ Packages', 'Software Package'

Answer: Keep things simple as you don't need a regex here, just use two boolean masks and a case independent comparison: # does the column name contain "software"? m1 = df.columns.str.contains('software', case=False) # does it contain "package"? m2 = df.columns.str.contains('package', case=False) # if both conditions are met, keep the column out = df.loc[:, m1&m2]  Example input: df = pd.DataFrame(columns=['Local Software Packages', 'Software XYZ Packages', 'Software Package', 'Other', 'Software only'], index=[0])  Output:  Local Software Packages Software XYZ Packages Software Package 0 NaN NaN NaN  If you just want the names: df.columns[m1&m2] # Index(['Local Software Packages', 'Software XYZ Packages', 'Software Package'], dtype='object')
Question: Running Elastic Net simulations by varying a couple parameters and looking to save output coefficients to a dataframe for potential review later. Ultimately looking to save off a dataframe with two parameter identifier columns (ie, 'alpha', 'l1_ratio') and a number of other columns for the resulting coefficients for the model fit with those parameters. 'alpha' is a float (varying from .1 to 1000 in increments) and 'l1_ratio' is a float from 0 to 1. 'coefs' is a numpy array that I'd like to expand into individual columns for each coefficient value (total number will stay fixed, say 5 for this simple case). For instance: alpha = .1 l1_ratio = .5 coefs = array([-1.30, -0.45, .04, .65, -0.88])  would result in a row record in the final dataframe of:  alpha l1_ratio c1 c2 c3 c4 c5 0 .1 .5 -1.30 -0.45 .04 .65 -0.88  I'll ultimately loop over and place additional rows for each scenario. Would also prefer not to label coefficient columns manually as there can be dozens depending on the situation--leaving column header empty is fine. How would I do this?

Answer: I imagine you will generate all data points iteratively. However, DataFrames don't like to be grown this way. Performance of adding new rows repeatedly is terrible. Assuming logic the function that will produce the new data points, I would use a dictionary to collect them, and create the DataFrame once in the end: data = {} for alpha, l1_ratio, coeffs in logic(): data[(alpha, l1_ratio)] = coeffs df = (pd.DataFrame(data).T .rename(columns=lambda x: f'c{x+1}') .rename_axis(['alpha', 'l1_ratio']) .reset_index() # optional )  Variant to generate the dataframe: df = (pd.DataFrame(data.keys(), columns=['alpha', 'l1_ratio']) .join(pd.DataFrame(data.values()) .rename(columns=lambda x: f'c{x+1}')) )  Example output:  alpha l1_ratio c1 c2 c3 c4 c5 0 0.1 0.5 -1.3 -0.45 0.04 0.65 -0.88 1 0.2 0.5 -1.3 -0.45 0.04 0.65 -0.88
Question: Project Euler problem 118 reads, "Using all of the digits 1 through 9 and concatenating them freely to form decimal integers, different sets can be formed. Interestingly with the set {2,5,47,89,631} all of the elements belonging to it are prime. How many distinct sets containing each of the digits one through nine exactly once contain only prime elements." This is the only problem I've encountered so far that doesn't provide an answer to an easier version of the question. As a result, it is difficult for me to check the algorithm I've made. Do you see what could be the problem? I coded up a pretty straightforward dynamic programming approach. For that, I will answer the question, "How many distinct sets containing each of the digits in S exactly once contain only prime elements", for different values of S, and then combine their results to answer the original question. First, I generate a list of all the relevant primes. This is every prime with 9 or fewer digits, that has no duplicate digits, and that doesn't have a 0 as one of its digits. I then use this list to create a set count dictionary. The keys to this dictionary are frozensets that represent which digits each prime has, and the values are integers that count the number of relevant primes that reduce to that set: from itertools import chain, combinations from functools import cache from collections import defaultdict from pyprimesieve import primes primes = [ p for p in primes(10**9) if len(set(str(p))) == len(str(p)) and "0" not in str(p) ] set_counts = defaultdict(int) for p in primes: set_counts[frozenset(map(int, str(p)))] += 1  set_counts will then serve as the base case for our recursion. Next, we need a way to break down our problem into relevant subproblems. So I wrote a function which will generate all of the ways to split a set into two disjoint nonempty subsets: @cache def set_decomps(s): """ Generates all possible ways to partition a set s into two non-empty subsets """ l = list( map( lambda x: (frozenset(x), s - frozenset(x)), chain.from_iterable(combinations(s, r) for r in range(1, len(s))), ) ) return l[: len(l) // 2]  Finally, we should be able to just use a simple memoized approach to use these together to solve the problem, and all of its subvariants. @cache def dp(s): """ Returns the number of ways to create a set containing prime numbers Such that the set of all the digits in the primes is equal to s With no duplicates or extra digits """ base = set_counts[s] if len(s) > 1: for a, b in set_decomps(s): base += dp(a) * dp(b) return base print(dp(frozenset(range(1,10)))) # prints 114566, which is wrong  Obviously there is a flaw in my approach.

Answer: The flaw is this. Consider the set {2,5,47,89,631}. Different initial splits can lead to it. One is {1,2,3,6}, {4,5,7,8,9} and another is {1,3,6,8,9} {2,4,5,7}. There are many more. And therefore you are overcounting. To leave you the fun of the problem, I won't tell you how to fix this overcounting. I'll just tell you that if you are counting multiple ways to get to a particular set, then your number will be too high.
Question: I am very confused by the following code: import dataclasses @dataclasses.dataclass() class Base(): x: int = 100 @dataclasses.dataclass() class Derived(Base): x: int = 200 @dataclasses.dataclass() class DerivedRaw(Base): x = 300 base = Base() derived = Derived() derived_raw = DerivedRaw() print(base.x) print(derived.x) print(derived_raw.x)  What it prints is: 100 200 100  I don't understand why the last number isn't 300. Why do the annotations matter? This seems to be an interaction with @dataclasses.dataclass(), as the code: class Base(): x: int = 100 class DerivedRaw(Base): x = 300 derived_raw = DerivedRaw() print(derived_raw.x)  Does print 300.

Answer: From the dataclass documentation:  The @dataclass decorator examines the class to find fields. A field is defined as a class variable that has a type annotation. ...  So to have proper dataclass field must have type annotation.
Question: I wrote this shopping program myself and I'm trying to get it to print to the shell and to the text file with each shopping list item on a new line. Please can you help me with this? #By Simeon Beckford-Tongs BSc MSc Copyright © 2024. All rights reserved. By Simeon Beckford-Tongs BSc MSc Copyright © 2024. All rights reserved. items = input('What is the first item on your shopping list:\n') items += input('What is the second item on your shopping list:\n') items += input('What is the third item on your shopping list:\n') items += input('What is the fourth item on your shopping list:\n') items += input('What is the fith item on your shopping list:\n') items += input('What is the sixth item on your shopping list:\n') items += input('What is the seventh item on your shopping list:\n') items += input('What is the eighth item on your shopping list:\n') items += input('What is the ninth item on your shopping list:\n') items += input('What is the tenth item on your shopping list:\n') shopping_list = 'shopping list:' file = open('shopping_list.txt', 'w') file.write (items) file.close() file = open('shopping_list.txt', 'r') for line in file: print( line, end = '\n') file.close()  I tried adding a \n character(s) to lines 3-12 and line 26.

Answer: So, you are appending each item to the items variable, but you're not adding any newline characters (\n) between them, which results in them being concatenated together into one long string. items = "" items += input('What is the first item on your shopping list:\n') + '\n' items += input('What is the second item on your shopping list:\n') + '\n' items += input('What is the third item on your shopping list:\n') + '\n' items += input('What is the fourth item on your shopping list:\n') + '\n' items += input('What is the fifth item on your shopping list:\n') + '\n' items += input('What is the sixth item on your shopping list:\n') + '\n' items += input('What is the seventh item on your shopping list:\n') + '\n' items += input('What is the eighth item on your shopping list:\n') + '\n' items += input('What is the ninth item on your shopping list:\n') + '\n' items += input('What is the tenth item on your shopping list:\n') + '\n' shopping_list = 'shopping list:' file = open('shopping_list.txt', 'w') file.write(items) file.close() file = open('shopping_list.txt', 'r') for line in file: print(line, end='\n') file.close()  This should to do what you asked for.
Question: I am using pre-trained LLM to generate a representative embedding for an input text. But it is wired that the output embeddings are all the same regardless of different input texts. The codes: from transformers import pipeline, AutoTokenizer, AutoModel import numpy as np PRETRAIN_MODEL = 'mistralai/Mistral-7B-Instruct-v0.2' tokenizer = AutoTokenizer.from_pretrained(PRETRAIN_MODEL) model = AutoModel.from_pretrained(PRETRAIN_MODEL) def generate_embedding(document): inputs = tokenizer(document, return_tensors='pt') print("Tokenized inputs:", inputs) with torch.no_grad(): outputs = model(**inputs) embedding = outputs.last_hidden_state[0, 0, :].numpy() print("Generated embedding:", embedding) return embedding text1 = "this is a test" text2 = "this is another test" text3 = "there are other tests" embedding1 = generate_embedding(text1) embedding2 = generate_embedding(text2) embedding3 = generate_embedding(text3) are_equal = np.array_equal(embedding1, embedding2) and np.array_equal(embedding2, embedding3) if are_equal: print("The embeddings are the same.") else: print("The embeddings are not the same.")  The printed tokens are different, but the printed embeddings are the same. The outputs: Tokenized inputs: {'input_ids': tensor([[ 1, 456, 349, 264, 1369]]), 'attention_mask': tensor([[1, 1, 1, 1, 1]])} Generated embedding: [-1.7762679 1.9293272 -2.2413437 ... 2.6379988 -3.104867 4.806004 ] Tokenized inputs: {'input_ids': tensor([[ 1, 456, 349, 1698, 1369]]), 'attention_mask': tensor([[1, 1, 1, 1, 1]])} Generated embedding: [-1.7762679 1.9293272 -2.2413437 ... 2.6379988 -3.104867 4.806004 ] Tokenized inputs: {'input_ids': tensor([[ 1, 736, 460, 799, 8079]]), 'attention_mask': tensor([[1, 1, 1, 1, 1]])} Generated embedding: [-1.7762679 1.9293272 -2.2413437 ... 2.6379988 -3.104867 4.806004 ] The embeddings are the same.  Does anyone know where the problem is? Many thanks!

Answer: You're not slicing it the dimensions right at outputs.last_hidden_state[0, 0, :].numpy()  Q: What is the 0th token in all inputs? A: Beginning of sentence token (BOS) Q: So that's the "embeddings" I'm slicing is the BOS token? A: Try this: from transformers import pipeline, AutoTokenizer, AutoModel import numpy as np PRETRAIN_MODEL = 'mistralai/Mistral-7B-Instruct-v0.2' tokenizer = AutoTokenizer.from_pretrained(PRETRAIN_MODEL) model = AutoModel.from_pretrained(PRETRAIN_MODEL) model(**tokenizer("", return_tensors='pt')).last_hidden_state  [out]: tensor([[[-1.7763, 1.9293, -2.2413, ..., 2.6380, -3.1049, 4.8060]]], grad_fn=<MulBackward0>)  Q: Then, how do I get the embeddings from a decoder-only model? A: Can you really get an "embedding" from a decoder-only model? The model outputs a hidden state per token it "regress" through, so different texts get different tensor output size. Q: How do you make it into a single fixed size vector then? A: Most probably,  some sort of pooling function? an open research question (as of now Apr 2024) but there's work on tools like https://arxiv.org/abs/2404.05961
Question: I have a dataframe with a column consisting of strings. I want to trim the strings in the column such that everything is removed after the first appearance of a given word. The words are in this list: words_to_trim_after = ['test', 'hello', 'very good']  So if I have a dataframe such as the following df = pd.DataFrame({'a':['test this is a test bla bla', 'hello bla bla this is a test', 'very good qwerty this is nice']})  I want to end up with df_end = pd.DataFrame({'a':['test', 'hello', 'very good']}) 

Answer: You can use split def trim_after_first_word(s, words): for word in words: parts = s.split(word, maxsplit=1) if len(parts) > 1: return word return None
Question: I have a dataframe with this structure    Order Number Line Number Item Type     12345 1 1001 Parent   12345 2 1002 Child   12345 3 1003 Child   12345 4 1004 Child   12345 5 1005 Parent   12345 6 1006 Child    I would like to add a column which shows the "Parent Item" for each item. The parent item is the first parent type that each child follows. There are no relationships or links to use. Line Number dictates the children for each parent.    Line Number Item Type Parent Item     1 1001 Parent 1001   2 1002 Child 1001   3 1003 Child 1001   4 1004 Child 1001   5 1005 Parent 1005   6 1006 Child 1005    The parent item number must repeat until a new parent is found. I have tried adding a LAG column to do checks but couldn't quite nail down the logic. I felt like I needed more than one column but couldn't do it. I've also tried a window function to "group" them together by row number, partitioning by order number and type but that doesn't work as it separates the parents from the children.

Answer: Try this: from pyspark.sql import functions as F from pyspark.sql.window import Window df = df.withColumn( "Parent_Item", F.last(F.when(F.col("Type") == "Parent", F.col("Item")), ignorenulls=True).over( Window.partitionBy("Order Number").orderBy("Line Number") ), ) df.show()  Output: +------------+-----------+----+------+-----------+ |Order Number|Line Number|Item| Type|Parent_Item| +------------+-----------+----+------+-----------+ | 12345| 1|1001|Parent| 1001| | 12345| 2|1002| Child| 1001| | 12345| 3|1003| Child| 1001| | 12345| 4|1004| Child| 1001| | 12345| 5|1005|Parent| 1005| | 12345| 6|1006| Child| 1005| +------------+-----------+----+------+-----------+