eshangj/stackoverflow_q_and_a_sample huggingface dataset:
Question: I am trying to do an url encoding on the following string the following string 'https://bla/ble:bli~' using the urllib standard library on Python 3.10. Problem is that the ~ character is not encoded to %7E as per url encoding. Is there a way around it? I have tried from urllib.parse import quote input_string = 'https://bla/ble:bli~' url_encoded = quote(input_string)  url_encoded takes the value 'https%3A//bla/ble%3Abli~' I have also tried: url_encoded = quote(input_string,safe='~')  In this case the url_encoded takes the value 'https%3A%2F%2Fbla%2Fble%3Abli~'

Answer: '~' character is not required to be encoded. (Though its encoded formate is %7E). The tilde is one of the allowed characters in url. So you can use any of the followings: from urllib.parse import quote input_string = 'https://bla/ble:bli~' url_encoded = quote(input_string)  or you can use request module as well import requests input_string = 'https://bla/ble:bli~' url_encoded = requests.utils.requote_uri(input_string)  Edit: As you are specific about url_encoding for all the special characters, I'm adding the following function. Use the following function: def url_encode(url): #add more characters into the list if required lst = ['[', '@', '_', '!', '#', '$', '%', '^', '&', '*', '(', ')', '<', '>', '?', '/', '\\', '|', '{', '}', '~', ':', ']]'] for x in url: if x in lst: new = hex(ord(x)).replace('x', '').upper() new = f'%{new}' url = url.replace(x, new) return url
Question: I'm curious about the result of this Python code that does a split on an empty string '': import re x = re.split(r'\W*', '') y = re.split(r'(\W*)', '')  Since the string is an empty string, I expect the result for x = re.split(r'\W*', '') is an empty list and that for y = re.split(r'(\W*)', '') is ['']. The actual result for x = re.split(r'\W*','')is ['',''] and that for y = re.split(r'(\W*)','') is ['','','']. I don't know what leads to these results.

Answer: Note that the regular expression \W* can match an empty string. Thus, while it's not useful, it's true that the empty string can be split in half to produce an empty string: '' = '' + '' + ''   The '' that precedes the regular expression The '' that matches the regular expression The '' that follows the regular expression  In the first case, you get strings 1 and 3. In the second case, you also get string 2. (In general, it's probably never a good idea to use a regular expression that can mach the empty string as the first argument.)
Question: I want to create a new column that contains the max, min and average of the next X elements. After searching for a while the only solution I was able to find was: df = df.with_columns( pl.col("price").shift(-20).rolling_max(window_size=20).alias('max'), pl.col("price").shift(-20).rolling_min(window_size=20).alias('min') )  However for the first 20 rows there is no data and results in nulls because the .rolling_max function applies the windows of the previous rows and not the next. Is there a better way to achieve this?

Answer: @RomanPekar's solution works well for the creation of few columns. However, when working with multiple columns it could be preferable to use a full forward-looking rolling context using pl.DataFrame.group_by_dynamic as follows. Data. df = pl.DataFrame({ "price": [1, 2, 6, 1, 3, 5, 8, 7], })  Usage of pl.DataFrame.group_by_dynamic. I've added multiple aggregations of the rolling window and a separate window column to clarify the approach. ( df .group_by_dynamic( index_column=pl.int_range(0, pl.len()), every="1i", period="2i", offset="0i", ) .agg( pl.col("price").first(), pl.col("price").alias("window"), pl.col("price").max().alias("min"), pl.col("price").max().alias("max"), ) )  Output. shape: (8, 5) ┌─────────┬───────┬───────────┬─────┬─────┐ │ literal ┆ price ┆ window ┆ min ┆ max │ │ --- ┆ --- ┆ --- ┆ --- ┆ --- │ │ i64 ┆ i64 ┆ list[i64] ┆ i64 ┆ i64 │ ╞═════════╪═══════╪═══════════╪═════╪═════╡ │ 0 ┆ 1 ┆ [1, 2] ┆ 2 ┆ 2 │ │ 1 ┆ 2 ┆ [2, 6] ┆ 6 ┆ 6 │ │ 2 ┆ 6 ┆ [6, 1] ┆ 6 ┆ 6 │ │ 3 ┆ 1 ┆ [1, 3] ┆ 3 ┆ 3 │ │ 4 ┆ 3 ┆ [3, 5] ┆ 5 ┆ 5 │ │ 5 ┆ 5 ┆ [5, 8] ┆ 8 ┆ 8 │ │ 6 ┆ 8 ┆ [8, 7] ┆ 8 ┆ 8 │ │ 7 ┆ 7 ┆ [7] ┆ 7 ┆ 7 │ └─────────┴───────┴───────────┴─────┴─────┘  This approach also allows a separate by column to perform these aggregations within groups defined by a separate column.
Question: I am attempting to italicize text in my plot: import matplotlib.pyplot as plt plt.plot([1,2,3],[1,1,1], label = '$\it{ABC123}$') plt.legend() plt.show()  as shown by Styling part of label in legend in matplotlib but this only italicized ABC, not 123, which was also noticed, but left unsolved, by Matplotlib italic font cannot be applied to numbers in the legend? I have also tried usetex italic symbols in matplotlib? but that changes the fonts which make this figure incompatible with other figures that I'm making. I'm on Python 3.10.12 and matplotlib 3.8.1 matplotlib-inline 0.1.3  how can I italicize both letters and numbers, e.g. in ABC123?

Answer: You can use the font_manager: import matplotlib.pyplot as plt import matplotlib.font_manager as font_manager font = font_manager.FontProperties(style='italic') plt.plot([1,2,3],[1,1,1], label = 'ABC123') plt.legend(prop=font) plt.show()   To use it for x/y ticks or labels, you must use the fontproperties argument: plt.yticks(fontproperties=font)
Question: Let's say I want to create an abstract base class called Document. I want the type checker to guarantee that all its subclasses implement a class method called from_paragraphs, which constructs a document from a sequence of Paragraph objects. However, a LegalDocument should only be constructable from LegalParagraph objects, and an AcademicDocument - only from AcademicParagraph objects. My instinct is to do it like so: from abc import ABC, abstractmethod from typing import Sequence class Document(ABC): @classmethod @abstractmethod def from_paragraphs(cls, paragraphs: Sequence["Paragraph"]): pass class LegalDocument(Document): @classmethod def from_paragraphs(cls, paragraphs: Sequence["LegalParagraph"]): return # some logic here... class AcademicDocument(Document): @classmethod def from_paragraphs(cls, paragraphs: Sequence["AcademicParagraph"]): return # some logic here... class Paragraph: text: str class LegalParagraph(Paragraph): pass class AcademicParagraph(Paragraph): pass  However, Pyright complains about this because from_paragraphs on the derived classes violates the Liskov substitution principle. How do I make sure that each derived class implements from_paragraphs for some kind of Paragraph?

Answer: Turns out this can be solved using generics: from abc import ABC, abstractmethod from typing import Generic, Sequence, TypeVar ParagraphType = TypeVar("ParagraphType", bound="Paragraph") class Document(ABC, Generic[ParagraphType]): @classmethod @abstractmethod def from_paragraphs(cls, paragraphs: Sequence[ParagraphType]): pass class LegalDocument(Document["LegalParagraph"]): @classmethod def from_paragraphs(cls, paragraphs): return # some logic here... class AcademicDocument(Document["AcademicParagraph"]): @classmethod def from_paragraphs(cls, paragraphs): return # some logic here... class Paragraph: text: str class LegalParagraph(Paragraph): pass class AcademicParagraph(Paragraph): pass  Saying bound="Paragraph" guarantees that the ParagraphType represents a (subclass of) Paragraph, but the derived classes are not expected to implement from_paragraphs for all paragraph types, just for the one they choose. The type checker also automatically figures out the type of the argument paragraphs for LegalDocument.from_paragraphs, saving me some work :)
Question: Does python's concurrent.futures.wait() preserve the order of job submission? I submitted two jobs to ThreadPoolExecutor as follows: import concurrent.futures import time, random def mult(n): time.sleep(random.randrange(1,5)) return f"mult: {n * 2}" def divide(n): time.sleep(random.randrange(1,5)) return f"divide: {n // 2}" with concurrent.futures.ThreadPoolExecutor() as executor: mult_future = executor.submit(mult, 200) divide_future = executor.submit(divide, 200) # wait for both task to complete mult_task, divide_task = concurrent.futures.wait( [mult_future, divide_future], return_when=concurrent.futures.ALL_COMPLETED, ).done mult_result = mult_task.result() divide_result = divide_task.result() print(mult_result) print(divide_result)  Sometimes I see divide: 50 mult: 400  and sometimes, mult: 400 divide: 50  shouldn't mult_task, divide_task always map to mult_future, divide_future ? python --version >> Python 3.8.16 

Answer: According to the doc that the function wait():  Returns a named 2-tuple of sets.  By definition, sets are unordered. That means you cannot guarantee the order. Since you already have mult_future and divide_future, you can use them to guarantee the order. There is no need to call wait either. import concurrent.futures import random import time def mult(n): time.sleep(random.randrange(1, 5)) return f"mult: {n * 2}" def divide(n): time.sleep(random.randrange(1, 5)) return f"divide: {n // 2}" with concurrent.futures.ThreadPoolExecutor() as executor: mult_future = executor.submit(mult, 200) divide_future = executor.submit(divide, 200) # Print in order: mult div print(mult_future.result()) print(divide_future.result())
Question: I am attempting to parse strings using Regex. The strings look like: Stack;O&verflow;i%s;the;best!  I want to parse it to: Stack&verflow%sbest!  So when we see a ; remove everything up until we see one of the following characters: [;,)%&@] (or replace with empty space ""). I am using re package in Python: string = re.sub('^[^-].*[)/]$', '', string)  This is what I have right now: ^[^;].*[;,)%&@]  Which as I understand it says: starting at the pattern with ;, read everything that matches in between ; and [;,)%&@] characters But the result is wrong and looks like: Stack;O&verflow;i%s;the;  Demo here. What am I missing? EDIT: @InSync pointed out that there is a discrepancy if ; is in the end characters as well. As worded above, it should result inStack&verflow%s**;**best! but instead I want to see Stack&verflow%sbest!. Perhaps two regex lines are appropriate here, I am not sure; if you can get to Stack&verflow%s**;**best! then the rest is just simple replacement of all the remaining ;. EDIT2: The code I found that works was import re def remove_semicolons(name): name = re.sub(';.*?(?=[;,)%&@])', '', name) name = re.sub(';','',name) return name remove_semicolons('Stack;O&verflow;i%s;the;best!')  Or if you feel like causing a headache to the next programmer who looks at your code: import re semicolon_string = 'Stack;O&verflow;i%s;the;best!' cleaned_string = re.sub(';','',re.sub(';.*?(?=[;,)%&@])', '', semicolon_string)) 

Answer: Alright in my answer I assume you have a typo in your expected output. Remove everything starting with ; up to (;,)%&@) and so Stack ;O &verflow ;i %s ;the ;best! would become Stack&verflow%s;best! for the regex you want to start with ; then anything after 0 or more times .* (if you require a character change to .+) followed by your ending characters [;,)%&@]. To exclude them you need to add a positive lookahead ?(?=[;,)%&@]). This as the name suggests looks ahead one character and tries to match it to your sequence For a final regex: ;.*?(?=[;,)%&@])  or if you require characters in between: ;.+?(?=[;,)%&@])
Question: I have a large (~300M rows x 44 cols) dataframe and I need to fill in null values in certain ways depending on the characteristics of each group. For example, say we have lf = pl.LazyFrame( {'group':(1,1,1,2,2,2,3,3,3), 'val':('yes', None, 'no', '2', '2', '2', 'answer', None, 'answer') } )  ┌───────┬────────┐ │ group ┆ val │ │ --- ┆ --- │ │ i64 ┆ str │ ╞═══════╪════════╡ │ 1 ┆ yes │ │ 1 ┆ null │ │ 1 ┆ no │ │ 2 ┆ 2 │ │ 2 ┆ 2 │ │ 2 ┆ 2 │ │ 3 ┆ answer │ │ 3 ┆ null │ │ 3 ┆ answer │ └───────┴────────┘  I want to fill in nulls if and only if the group contains a single non-null unique value in the other cells, since in my context that's the expectation of the data and the presense of more than one unique value (or all nulls) in the group signals another issue that will be handled differently. I'm able to fill null values for each group with the following: filled_lf = ( lf .with_columns( pl.col('val') .fill_null(pl.col('val').unique().first().over('group')).alias('filled_val') ) )  However, for one, it seems that pl.col('val').unique() includes 'null' as one of the values, and the ordering is stochastic so choosing the first value on the list has inconsistent results. Secondly, it doesn't include the condition I need. Desired result: ┌───────┬────────┬────────────┐ │ group ┆ val ┆ filled_val │ │ --- ┆ --- ┆ --- │ │ i64 ┆ str ┆ str │ ╞═══════╪════════╪════════════╡ │ 1 ┆ yes ┆ yes │ │ 1 ┆ null ┆ null │ │ 1 ┆ no ┆ no │ │ 2 ┆ 2 ┆ 2 │ │ 2 ┆ 2 ┆ 2 │ │ 2 ┆ 2 ┆ 2 │ │ 3 ┆ answer ┆ answer │ │ 3 ┆ null ┆ answer │ │ 3 ┆ answer ┆ answer │ └───────┴────────┴────────────┘  Pandas 3.12 Polars 0.20.1 Thanks in advance for your advice!

Answer: You can add:  .drop_nulls() .unique() with maintain_order=True when/then/otherwise to implement the conditional count/len logic  unique = pl.col("val").drop_nulls().unique(maintain_order=True) df.with_columns( pl.when(unique.len().over("group") == 1) .then(pl.col("val").fill_null(unique.first().over("group"))) .otherwise(pl.col("val")) .alias("filled") )  shape: (9, 3) ┌───────┬────────┬────────┐ │ group ┆ val ┆ filled │ │ --- ┆ --- ┆ --- │ │ i64 ┆ str ┆ str │ ╞═══════╪════════╪════════╡ │ 1 ┆ yes ┆ yes │ │ 1 ┆ null ┆ null │ │ 1 ┆ no ┆ no │ │ 2 ┆ 2 ┆ 2 │ │ 2 ┆ 2 ┆ 2 │ │ 2 ┆ 2 ┆ 2 │ │ 3 ┆ answer ┆ answer │ │ 3 ┆ null ┆ answer │ │ 3 ┆ answer ┆ answer │ └───────┴────────┴────────┘
Question: I want use numba for scipy.fsolve: from scipy.optimize import fsolve from numba import njit @njit def FUN12(): XGUESS=[8.0,7.0] X =[0.0,0.0] try: X = fsolve(FCN3, XGUESS) except: print("error") return X @njit def FCN3(X): F=[0.0,0.0] F[0]=4.*pow(X[0],2)-3.*pow(X[1],1)-7 F[1] = 5.*X[0] -2. * pow(X[1] , 2)+8 return F FUN12()  I got this error for my code: numba.core.errors.TypingError: Failed in nopython mode pipeline (step: nopython frontend) Untyped global name 'fsolve': Cannot determine Numba type of <class 'function'>

Answer: Numba compatible code is only some subset of Python and Numpy functions. You should not run fsolve inside numba function. You should remove njit decorator from FUN12 function. You can keep it for FCN3. from scipy.optimize import fsolve from numba import njit # @njit remove this line, def FUN12(): XGUESS=[8.0,7.0] X =[0.0,0.0] try: X = fsolve(FCN3, XGUESS) except: print("error") return X @njit def FCN3(X): F=[0.0,0.0] F[0]=4.*pow(X[0],2)-3.*pow(X[1],1)-7 F[1] = 5.*X[0] -2. * pow(X[1] , 2)+8 return F FUN12()
Question: What is the difference between polars.read_csv vs polars.read_csv_batched vs polars.scan_csv ? polars.read_csv looks equivalent to pandas.read_csv as they have the same name. Which one to use in which scenario and how they are similar/different to pandas.read_csv?

Answer:  polars.read_csv_batched is pretty equivalent to pandas.read_csv(iterator=True).  polars.scan_csv doesn't do anything until you perform an operation on the dataframe like dask.dataframe.read_csv (lazy loading).   Scenarios:  I use pandas.read_csv when my data is messy or complex in structure and the data is not too large  I use polars.read_csv when my data file is very large (> 10GB).   This is an answer based solely on my (humble) opinion.
Question: Hi I am trying to merge two dataset in the following way: df1=pd.DataFrame({'company name':['A','B','C'], 'analyst 1 name':['Tom','Mike',np.nan], 'analyst 2 name':[np.nan,'Alice',np.nan], 'analyst 3 name':['Jane','Steve','Alex']}) df2=pd.DataFrame({'company name':['A','B','C'], 'score 1':[3,5,np.nan], 'score 2':[np.nan,1,np.nan], 'score 3':[6,np.nan,11]}) df_desire=pd.DataFrame({'company name':['A','A','B','B','B','C'], 'analyst':['Tom','Jane','Mike','Alice','Steve','Alex'], 'score':[3,6,5,1,np.nan,11]})  Basically, df1 contains analyst name and df2 contains scores that analysts assign. I am trying to merge two into df_desire. The way to read two tables is: for company A, two person cover it, which are Tom and Jane, who assign 3 and 6 respectively. Noted that even though Steve covers company B but I deliberately assign the score to be NA for robustness purpose. What I have done is : pd.concat([df1.melt(id_vars='company name',value_vars=['analyst 1 name','analyst 2 name','analyst 3 name']),\ df2.melt(id_vars='company name',value_vars=['score 1','score 2','score 3'])],axis=1)  I am looking for a more elegant solution.

Answer: Try: x = ( df1.set_index("company name") .stack(dropna=False) .reset_index(name="name") .drop(columns="company name") ) y = df2.set_index("company name").stack(dropna=False).reset_index(name="score") print( pd.concat([x, y], axis=1)[["company name", "name", "score"]] .dropna(subset=["name", "score"], how="all") .reset_index(drop=True) )  Prints:  company name name score 0 A Tom 3.0 1 A Jane 6.0 2 B Mike 5.0 3 B Alice 1.0 4 B Steve NaN 5 C Alex 11.0
Question: In a project I am working with duckdb to perform some queries on dataframes. For one of the queries, I have some user-input that I need to add to the query. That is why I am wondering if SQL-Injection is possible in this case. Is there a way a user could harm the application or the system through the input? And if so, how could I prevent this case? It seems that duckdb has no PreparedStatement for queries on dataframes. I already looked up in the documentation (https://duckdb.org/docs/api/python/overview.html) but couldn't find anything useful. The method duckdb.execute(query, parameters) only seems to work on databases with a real sql-connection and not on dataframes. There is another question on stackoverflow (Syntax for Duckdb > Python SQL with Parameter\Variable) about this topic but the answer only works on real sql-connections and the version with f-strings seems insecure to me. Here is a small code sample to show what I mean: import duckdb import pandas as pd df_data = pd.DataFrame({'id': [1, 2, 3, 4], 'student': ['student_a', 'student_a', 'student_b', 'student_c']}) user_input = 3 # fetch some user_input here # How to prevent sql-injection, if its even possible in this case? result = duckdb.query("SELECT * FROM df_data WHERE id={}".format(user_input))  So how would you approach this problem? Is sql-injection even possible? Thanks for your help and feel free to ask for more details, if you need some more information! EDIT: Fixed a syntax error in the code

Answer:  The method duckdb.execute(query, parameters) only seems to work on databases with a real sql-connection and not on dataframes.  It seems it's possible: >>> duckdb.execute("""SELECT * FROM df_data WHERE id=?""", (user_input,)).df() id student 0 3 student_b
Question: Given a list of strings such as: foo = \['A', 'A', 'B', 'A', 'B', 'C', 'C', 'A', 'B', 'C', 'A'\]  How can we label them such that the output would be: output = \['A1', 'A2', 'B1', 'A3', 'B2', 'C1', 'C2', 'A4', 'B2', 'C3', 'A5'\]  (keeping the order of the original list) In the following case there are only 3 unique variables to look at, so the first think I tried was looking at the unique elements: import numpy as np np.unique(foo) Output = \['A', 'B', 'C'\]  But then I get stacked when trying to find the proper loop to reach the desired output.

Answer: Using pure python, take advantage of a dictionary to count the values: foo = ['A', 'A', 'B', 'A', 'B', 'C', 'C', 'A', 'B', 'C', 'A'] d = {} out = [] for val in foo: d[val] = d.get(val, 0)+1 out.append(f'{val}{d[val]}')  If you can use pandas: import pandas as pd s = pd.Series(foo) out = s.add(s.groupby(s).cumcount().add(1).astype(str)).tolist()  Output: ['A1', 'A2', 'B1', 'A3', 'B2', 'C1', 'C2', 'A4', 'B3', 'C3', 'A5']
Question: I am in the process of building a RAG like the one in this Video. However, I cannot import FAISS like this. from langchain.vectorstores import FAISS LangChainDeprecationWarning: Importing vector stores from langchain is deprecated. Importing from langchain will no longer be supported as of langchain==0.2.0. Please import from langchain-community instead: `from langchain_community.vectorstores import faiss`.  However, it is possible to import faiss: from langchain_community.vectorstores import faiss  But with this it is not possible to call faiss.from_text(). vectorstore = faiss.from_texts(text=text_chunks, embeddings=embeddings) AttributeError: module 'langchain_community.vectorstores.faiss' has no attribute 'from_texts'  Is it no longer possible to call .from_text() with the current one? I didn't find anything about this in the documentation. Python=3.10.13

Answer: The solution was for me to importing the FAISS class directly from the langchain.vectorstores.faiss module and then using the from_documents method. from langchain_community.vectorstores.faiss import FAISS  I had importing the faiss module itself, rather than the FAISS class from the langchain.vectorstores.faiss module. The from_documents method is a class method of the FAISS class, not a function of the faiss module.
Question: I have an Airflow DAG and I use {{ ds }} to get the logical date. As per Airflow documentation template {{ ds }} return logical date in format YYYY-MM-DD in string format. So I am using following code to manipulate the dates (datetime.strptime('{{ dag_run.logical_date|ds }}', '%Y-%m-%d') - timedelta(3)).strftime('%Y-%m-%d')  but getting following error  Broken DAG: [/usr/local/airflow/dags/custom_dags/feature_store_daily.py] Traceback (most recent call last): File "/usr/lib/python3.10/_strptime.py", line 568, in _strptime_datetime tt, fraction, gmtoff_fraction = _strptime(data_string, format) File "/usr/lib/python3.10/_strptime.py", line 349, in _strptime raise ValueError("time data %r does not match format %r" % ValueError: time data '{{dag_run.logical_date|ds}}' does not match format '%Y-%m-%d'  I am not able to figure out why I am getting this error. Please advice

Answer: Firstly I thought the problem could be de lack of spaces before and after the "|" operand, but I´ve tested and works fine. In this case, the main problem is that you´re not using the Jinja template where it is effectively parsed. Due to this, the error is that you don´t have a date to format, but a string as it is. task_xx = SomeOperator( ... a="{{ dag_run.logical_date|ds }}", b="{{dag_run.logical_date | ds}}", c="{{ dag_run.logical_date | ds }}" )  And then, in the SomeOperator these attributes must be in template_fields so it can be parsed: class SomeOperator(BaseOperator): template_fields = ['a', 'b', 'c'] ....  Now, when you use these attributes, you´ll get what you want. I´ve tested and works fine. Or, you can also use Jinja templates with fields that already accept this.
Question: I'm looking for the recommended way to select an individual row of a polars.DataFrame by row number: something largely equivalent to pandas.DataFrame's .iloc[[n]] method for a given integer n. For polars imported as pl and a polars DataFrame df, my current approach would be: # for example n = 3 # create row index, filter for individual row, drop the row index. new_df = ( df.with_row_index() .filter(pl.col('index') == n) .select(pl.exclude('index')) )  I'm migrating from Pandas, and I've read the Pandas-to-Polars migration guide, but a slick solution to this specific case wasn't addressed there. Edit: to clarify, I am looking for an approach that returns a polars.DataFrame object for the chosen row. Does anyone have something slicker?

Answer: This is a very good sheet by: @Liam Brannigan Credit to them. https://www.rhosignal.com/posts/polars-pandas-cheatsheet/ A glimse from the sheet:  You can find other information related to Filtering Rows using iloc and its equivalent in polars in the sheet.