eshangj/stackoverflow_q_and_a_sample huggingface dataset:
Question: Take this example: df = (polars .DataFrame(dict( j=polars.datetime_range(datetime.date(2023, 1, 1), datetime.date(2023, 1, 3), '8h', closed='left', eager=True), )) .with_columns( k=polars.lit(numpy.random.randint(10, 99, 6)), ) ) j k 2023-01-01 00:00:00 47 2023-01-01 08:00:00 22 2023-01-01 16:00:00 82 2023-01-02 00:00:00 19 2023-01-02 08:00:00 85 2023-01-02 16:00:00 15 shape: (6, 2)  Here, numpy.random.randint(10, 99, 6) uses hard-coded 6 as the height of DataFrame, so it won't work if I changed e.g. the interval from 8h to 4h (which would require changing 6 to 12). I know I can do it by breaking the chain: df = polars.DataFrame(dict( j=polars.datetime_range(datetime.date(2023, 1, 1), datetime.date(2023, 1, 3), '4h', closed='left', eager=True), )) df = df.with_columns( k=polars.lit(numpy.random.randint(10, 99, df.height)), ) j k 2023-01-01 00:00:00 47 2023-01-01 04:00:00 22 2023-01-01 08:00:00 82 2023-01-01 12:00:00 19 2023-01-01 16:00:00 85 2023-01-01 20:00:00 15 2023-01-02 00:00:00 89 2023-01-02 04:00:00 74 2023-01-02 08:00:00 26 2023-01-02 12:00:00 11 2023-01-02 16:00:00 86 2023-01-02 20:00:00 81 shape: (12, 2)  Is there a way to do it (i.e. reference df.height or an equivalent) in one chained expression though?

Answer: You can use .pipe() df = ( pl.datetime_range( datetime.date(2023, 1, 1), datetime.date(2023, 1, 3), "4h", closed="left", eager=True ) .alias("date") .to_frame() )  df.pipe(lambda df: df.with_columns(pl.lit(np.random.randint(10, 99, df.height)).alias("rand")) )  shape: (12, 2) ┌─────────────────────┬──────┐ │ date ┆ rand │ │ --- ┆ --- │ │ datetime[μs] ┆ i64 │ ╞═════════════════════╪══════╡ │ 2023-01-01 00:00:00 ┆ 39 │ │ 2023-01-01 04:00:00 ┆ 45 │ │ 2023-01-01 08:00:00 ┆ 95 │ │ 2023-01-01 12:00:00 ┆ 72 │ │ … ┆ … │ │ 2023-01-02 08:00:00 ┆ 34 │ │ 2023-01-02 12:00:00 ┆ 42 │ │ 2023-01-02 16:00:00 ┆ 30 │ │ 2023-01-02 20:00:00 ┆ 83 │ └─────────────────────┴──────┘  As for the example task, perhaps .sample() could be used. df.with_columns( pl.int_range(10, 100).sample(pl.len(), with_replacement=True).alias("rand") )  shape: (12, 2) ┌─────────────────────┬──────┐ │ date ┆ rand │ │ --- ┆ --- │ │ datetime[μs] ┆ i64 │ ╞═════════════════════╪══════╡ │ 2023-01-01 00:00:00 ┆ 25 │ │ 2023-01-01 04:00:00 ┆ 27 │ │ 2023-01-01 08:00:00 ┆ 68 │ │ 2023-01-01 12:00:00 ┆ 95 │ │ 2023-01-01 16:00:00 ┆ 96 │ │ … ┆ … │ │ 2023-01-02 04:00:00 ┆ 36 │ │ 2023-01-02 08:00:00 ┆ 25 │ │ 2023-01-02 12:00:00 ┆ 90 │ │ 2023-01-02 16:00:00 ┆ 92 │ │ 2023-01-02 20:00:00 ┆ 92 │ └─────────────────────┴──────┘
Question: I want to remove character accents from a text column, ex. convert Piña to Pina. This is how I would do it in pandas: (names .str.normalize('NFKD') .str.encode('ascii', errors='ignore') .str.decode('utf-8'))  Polars has str.decode and str.encode but they don't seem to be what i'm looking for. Thanks!

Answer: To expand on @jqurious's comment you can do one of two things:  map_elements/lambda  like this: from unicodedata import normalize df.with_columns( a=pl.col('a') .map_elements(lambda x: normalize('NFKD',x) .encode('ascii', errors='ignore') .decode('utf-8')))   define function/map_batches  like this: from unicodedata import normalize def custnorm(In_series): for i, x in enumerate(In_series): newvalue = normalize('NFKD',x).encode('ascii', errors='ignore').decode('utf-8') if newvalue != x: In_series[i]=newvalue return In_series  then inside the df you can do df.with_columns(a=pl.col('a').map_batches(custnorm))  The difference between map_elements and map_batches is that map_elements tells polars to do the looping one row at a time whereas map_batches tells polars to feed the whole column as a Series to the function which must then return a Series of the same size.
Question: Basically when to use torch.gather vs torch.index_select I have a scenario where I am using positional embedding (max_len, batch_size, embedding_dim). Here, I would like to select only particular indices from the max_len axis. I want the result to be (new_indices, batch_size, embedding_dim). Searching around I found two functions but there purpose seem to be the same. Can someone exaplain which will be ideal for my situation and why? I am just curious.

Answer: I made a post about this (because I also always kept forgetting). The jist is in this image, the full version is here if you're interested.
Question: I am getting the following error when trying to create an object with Selenium Webdriver. "\selenium\webdriver\common\driver_finder.py", line 42, in get_path path = SeleniumManager().driver_location(options) if path is None else path "\selenium\webdriver\common\selenium_manager.py", line 74, in driver_location browser = options.capabilities["browserName"] AttributeError: 'str' object has no attribute 'capabilities' During handling of the above exception, another exception occurred: Traceback (most recent call last): "\selenium_webdriver_webscraping.py", line 4, in <module> driver = webdriver.Chrome(chrome_driver_path) "\selenium\webdriver\chrome\webdriver.py", line 47, in __init__ self.service.path = DriverFinder.get_path(self.service, self.options) "\selenium\webdriver\common\driver_finder.py", line 44, in get_path raise NoSuchDriverException(f"Unable to obtain {service.path} using Selenium Manager; {err}") selenium.common.exceptions.NoSuchDriverException: Message: Unable to obtain chromedriver using Selenium Manager; 'str' object has no attribute 'capabilities'; For documentation on this error, please visit: https://www.selenium.dev/documentation/webdriver/troubleshooting/errors/driver_location  This is the code I used: from selenium import webdriver chrome_driver_path = <chrome drive .exe path> driver = webdriver.Chrome(chrome_driver_path) 

Answer: If the Selenium version you are using is v4.6.0 or above (which I think it is as I see SeleniumManger in the error trace), then you don't really have to set the driver.exe path. Selenium can handle the browser and drivers by itself. So your code can be simplified as below: from selenium import webdriver driver = webdriver.Chrome() driver.get("https://www.google.com/") driver.quit()  A few references:  Purpose of webdriver manager Introducing Selenium Manager
Question: An application I've hosted online throws an error whenever it tries to query a pinecone database that I've set up. Whenever I run the same code (same pinecone environment and API key) on my local device, the queries go through just fine. Any ideas on what could be causing this issue? urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='aamcdoc-fb22780.svc.northamerica-northeast1-gcp.pinecone.io', port=443): Max retries exceeded with url: /query (Caused by NewConnectionError('<urllib3.connection.HTTPSConnection object at 0x7fd50980ce20>: Failed to establish a new connection: [Errno 111] Connection refused')) Edit: I found this on PythonAnywhere's Q&A (it's the service I'm using to host the web application) in regards to a similar question: "It appears that you are not configuring your code to use the proxy on PythonAnywhere, so you cannot connect out from your free account. Have a look at the documentation for the library you're using to see how to configure it to use the proxy." If that's the error, how would I go about fixing it?

Answer: I faced the exact same thing. I just fixed it, also my first SO answer ever. Using Python 3.10, pinecone-client you have to pass their proxy server during Pinecone init. Like this from pinecone.core.client.configuration import Configuration as OpenApiConfiguration openapi_config = OpenApiConfiguration.get_default_copy() openapi_config.proxy = "http://proxy.server:3128" pinecone.init( api_key="XXXXXXXXXXXXXXXXXXXXXXXXxx", environment="XXXXXXXXXXXXXXX", openapi_config=openapi_config )
Question: Hello there thanks for reading my post. I was using this same code yesterday and it was okay but today stoped working and i got this error: raise Forbidden(response) tweepy.errors.Forbidden: 403 Forbidden When authenticating requests to the Twitter API v2 endpoints, you must use keys and tokens from a Twitter developer App that is attached to a Project. You can create a project via the developer portal.  This is the code I implemented: import tweepy import csv #Autorización bearer_token = "" client = tweepy.Client(bearer_token=bearer_token) #Get the user id user="" user_id= client.get_user(username=user).id # Get all the tweets of a user paginator = tweepy.Paginator( client.get_users_tweets, # The method you want to use user_id, # Some argument for this method #end_time=datetime.datetime(2022, 3, 3, 19, 4, 49), tweet_fields=["created_at", "public_metrics", "possibly_sensitive", "attachments", "in_reply_to_user_id"], max_results=100, # How many tweets per page limit=40 # How many pages to retrieve ) data=[] count=0 #Create a csv file to store the data filename="@"+user+"_twitterdata.csv" f = open(filename,'a', newline='') writer = csv.writer(f) #Store the data for tweet in paginator.flatten(limit=40000): # Total number of tweets to retrieve count=count+1 #Check if tha tweet has multimedia content if tweet.attachments!=None:media=True else:media=False #Check if tha tweet is a retweet Retweet=tweet.text[:2]=="RT" #Check if tha tweet is a repy Reply=tweet.in_reply_to_user_id!=None row=[tweet.id, tweet.created_at, tweet.public_metrics.get("retweet_count"), tweet.public_metrics.get("reply_count"), tweet.public_metrics.get("like_count"), tweet.public_metrics.get("quote_count"), tweet.public_metrics.get("impression_count"), tweet.possibly_sensitive, media, #tweet.text.replace(",", "" ), Retweet, Reply #tweet.context_annotations ] writer.writerow(row) print("Retrieved ",count," tweets") # Write the data and close file writer = csv.writer(f) f.close()  I tried to generate new keys in the developer portal but that didn't help. Also i tried to change the User autentication settings in the twitter developer portal but that didn't work

Answer: With the "free" plan of the Twitter API you can no longer lookup tweets, GET /2/tweets/:id is only available in the "basic" plan, see: https://developer.twitter.com/en/portal/products/free
Question: I have the following pandas dataframe import pandas as pd df = pd.DataFrame({'cl1': ['A','A','A','A', 'A','A','A','A', 'D','D','D','D', 'D','D','D','D'], 'cl2': ['C','C','C','C', 'B','B','B','B', 'C','C','C','C', 'B','B','B','B'], 'p1p2': ['00','01','10','11', '00','01','10','11', '00','01','10','11', '00','01','10','11'], 'val':[1,2,3,4, 10,20,30,40, 5,6,7,8, 50,60,70,80]}) df cl1 cl2 p1p2 val 0 A C 00 1 1 A C 01 2 2 A C 10 3 3 A C 11 4 4 A B 00 10 5 A B 01 20 6 A B 10 30 7 A B 11 40 8 D C 00 5 9 D C 01 6 10 D C 10 7 11 D C 11 8 12 D B 00 50 13 D B 01 60 14 D B 10 70 15 D B 11 80  And I would like to create a plot that looks like this  How could I do that in python ?

Answer: Assuming that p1p2 only contains values 00, 01, 10, or 11, it is easy to use pivot table to get something like this: d = df.copy() d['p1'] = d['p1p2'].str[0] d['p2'] = d['p1p2'].str[1] counts = d.pivot_table(values = 'val', columns = ['cl1', 'p1'], index = ['cl2', 'p2']) counts # cl1 A D # p1 0 1 0 1 # cl2 p2 # B 0 10 30 50 70 # 1 20 40 60 80 # C 0 1 3 5 7 # 1 2 4 6 8  As for the plotting, here's something that is pretty close to what you are looking for: from matplotlib import pyplot as plt import seaborn as sns import numpy as np from matplotlib.colors import ListedColormap colors = pd.pivot_table(d, values = 'p1p2', columns = ['cl1', 'p1'], index = ['cl2', 'p2']).astype(int) colors = colors.astype(str).applymap(lambda val: int(val, 2)) color_labels = pd.pivot_table(d, values = 'val', columns = ['cl1', 'cl2'], index = ['p1p2']).index counts.index = counts.index.droplevel(-1) counts.columns = counts.columns.droplevel(-1) colors.index = colors.index.droplevel(-1) colors.columns = colors.columns.droplevel(-1) cmap = ListedColormap(['C0', 'C1', 'C2', 'C3']) ax = sns.heatmap(colors, cmap = cmap, annot = counts, cbar_kws = {"label": "p1p2 (legend)"}) colorbar = colorbar = ax.collections[0].colorbar colorbar.set_ticks(np.linspace(0, 3, 9)[1::2]) colorbar.set_ticklabels(color_labels) plt.show()
Question: I have a df which looks like this: api_spec_id type_of_change label 213 Breaking NaN 213 Breaking major 213 Non-Breaking patch 345 Non-Breaking NaN 345 Non-Breaking patch 345 Non-Breaking patch 678 Breaking NaN 678 Breaking minor 678 Breaking major 123 Breaking NaN 123 Breaking NaN  I want to calculate the unique number of api_spec_id where: all type_of_change are breaking, where expected output would be 2 (ids: 678,123) all type_of_change are non-breaking, where expected output would be 1 (id:345) at least one type_of_change is breaking, expected output: 3 (ids:213,678,123) at least one type_of_change is non-breaking, expected output: 2 (ids:213,345) I am not sure how I can achieve this, any suggestions or ideas would be greatly appreciated.

Answer: A possible solution : from functools import partial grp = df.groupby("api_spec_id") def detect(g, how, change): if how == "all": return g["type_of_change"].eq(change).all() elif how == "any": return g["type_of_change"].eq(change).any() def get_id(df): return df["api_spec_id"].unique().tolist() v1 = grp.filter(partial(detect, how="all", change="Breaking")).pipe(get_id) v2 = grp.filter(partial(detect, how="all", change="Non-Breaking")).pipe(get_id) v3 = grp.filter(partial(detect, how="any", change="Breaking")).pipe(get_id) v4 = grp.filter(partial(detect, how="any", change="Non-Breaking")).pipe(get_id)  Output : print(v1) # [678, 123] print(v2) # [345] print(v3) # [213, 678, 123] print(v4) # [213, 345]
Question: My Python docstrings & block comments in Visual Studio Code always used to be a different color to the single line comment. They use to be:  docstrings & block comments: orange single line comments: green  I did a reinstall of Visual Studio Code this morning and the block comments and docstrings are now the same color as the single line comments (so all comments are green). I have pasted an image (in case the colors here are different).  I have always used the default dark mode (and never change the default settings). Has something changed or is there a setting that could change this back?

Answer: According to the Release notes for 1.79 (the pull request section), they changed the docstring comment colors with this pull request. You can set your own colors for this using the solution in the other answer.
Question: For pandas DataFrames in python, multiple member methods have an inplace parameter which purportedly allow you to NOT create a copy of the object, but rather to directly modify the original object*. [*Edited to add: however, this proves to not be the case as pointed out by @juanpa.arrivillaga. inplace=True DOES copy data and merely updates a pointer associated with the modified object, so has few advantages over a manual re-assignment to the name of the original object.] Examples that I have seen online for the use of inplace=True do not include examples where chaining is used. This comment in a related SO thread may be an answer to why I don't see such examples anywhere:  you can't method chain and operate in-place. in-place ops return None and break the chain  But, would "inplace chaining" work if you put an inplace=True in the last entry in the chain? [Edited to add: no] Or would that be equivalent to trying to change a copy created in an earlier link in the chain, which, as it is no longer your original object, is "lost" after the chain statement is complete? [Edited to add: yes; see answer here] The use of large data objects would seem to preclude the notion of chaining without the ability to do so in-place, at least insofar as desire to maintain a low memory overhead and high computational speed. Is there an alternate implementation of pandas or, e.g. an equivalent of R's data.table available in python that might be appropriate for my needs? Or are my only options to not chain (and compute quickly) or to chain but make redundant copies of the data, at least transiently?

Answer: Let's try it. import pandas as pd import numpy as np df = pd.DataFrame({'value' : [2, 2, 1, 1, 3, 4, 5, np.NaN]}) df.sort_values('value').drop_duplicates().dropna(inplace=True)  Expect:  value 2 1.0 0 2.0 4 3.0 5 4.0 6 5.0  Result:  value 0 2.0 1 2.0 2 1.0 3 1.0 4 3.0 5 4.0 6 5.0 7 NaN  Answer: No, inplace=True at the end of the chain does not modify the original dataframe.
Question: I am trying to test out some code in reference to Robert Heaton's article explaining the difference between the different concepts of passing parameters into functions. Why are list and myList stored in the same location in memory when, according to Heaton, they should be two completely separate variables. Here is what I have written: def main(): myList = [0] print(f"myList: {id(myList)}") print(f"[0]: {id([0])}") append(myList) print(f"After call: {myList}") def append(list): print(f"append's list: {id(list)}") list.append(1)  According to Heaton, with pass-by-object-reference "the function provides its own box and creates a new variable for itself". As such, I expected that id(myList) and id(list) should be two different variables stored in different locations in memory. However, the output I receive is: myList: 1941960670528 [0]: 1941963219200 append's list: 1941960670528 After call: [0, 1] 

Answer: You have to consider the entire paragraph. The article is not saying it should be 2 different objects. There is one list, but separate variables pointing to it. Consider the following (list variable changed to l because built-ins shouldn't be used for variable names.): def append(l): print(f"append's list: {id(l)}") l.append(1) l = [] print(f"append's list after reassigning: {id(l)}") myList = [0] print(f"myList: {id(myList)}") append(myList) print(f"After call: {myList}") print(f"myList id after call: {id(myList)}")  We reassigned l to a new list, and yet myList outside is still pointing to the list containing [0, 1] What it's essentially trying to show, is that there are 3 parts to a variable - a name, reference and its value. Names - e.g. l, myList - is what we refer to in code. They, in turn get resolved to a reference - to simplify, the value given by id() - to find a place in memory where the actual value is stored. So, when we pass a list to a function, same reference is given a new name we can use inside the function. Changing the value related to this reference will be visible to the outside function, as the reference is the same. If we do something that changes the reference, e.g. assign a new list with l = [], this will not affect the outside list, as it still uses the old reference. To summarise, we've got 2 relationships: name -> reference and reference -> value When calling a function, we give the reference a new name, resulting in something like: name_outer -\ >- reference -- value name_inner -/  Changing the value results in something like: name_outer -\ >- reference -- new_value name_inner -/  While changing the reference results in: name_outer -- reference -- value name_inner -- new_reference -- new_value
Question: I have a data frame in which there is a column containing several NaN values. The dataframe looks like this:  col_1 col_2 2022-10-31 99.094 102.498 2022-11-30 99.001 101.880 2022-12-31 NaN 108.498 2023-01-31 NaN 100.500  I want to fill those NaN based on the simple calculation below: desired_val = (previous value in col_1 * current value in col_2) / previous value in col_2 which means, df.loc['2022-12-31', 'col_1'] should be = (99.001 * 108.498) / 101.880 = 105.432 and df.loc['2023-01-31', 'col_1'] should be = (105.432 * 100.500) / 108.498 = 97.660 I found solution by using row by row operation but it is slow when the dataset is big. I tried column wise operation by using this: df['col_1'] = df['col_1'].fillna( (df[col_1].shift(1) * df[col_2]) / df[col_2].shift(1) )  But it does work only for one row and then it does not go further. Is there any column wise pandas solution for that?

Answer: You can think of your operation and see that you multiply by x in one row and divide by x in the next row. Thus you can simplify the result to: col1_value = (last_valid_col1_value * current_col2_value) / col2_value_at_last_valid_col1_position  Which can be translated as: # is the row a NA? m1 = df['col_1'].isna() # is the next row a NA? m2 = df['col_1'].shift(-1).isna() df.loc[m1, 'col_1'] = (df['col_1'].div(df['col_2']) .where(m2 & ~m1).ffill() .mul(df['col_2'])[m1] )  Output:  col_1 col_2 2022-10-31 99.094000 102.498 2022-11-30 99.001000 101.880 2022-12-31 105.431984 108.498 2023-01-31 97.659997 100.500  Intermediates:  col_1 col_2 m1 m2 m2&~m1 ffilled(col1/col2) result 2022-10-31 99.094 102.498 False False False NaN NaN 2022-11-30 99.001 101.880 False True True 0.971741 NaN 2022-12-31 NaN 108.498 True True False 0.971741 105.431984 2023-01-31 NaN 100.500 True True False 0.971741 97.659997
Question: How do I read in a .csv file in Python with columns of varying lengths? I want to create a dictionary from the .csv file, with the .csv columns as lists of dictionary values. I've figured out how to write the dictionary to a .csv file, but I need help reading in that same file. import csv import itertools path = 'C:/Users/.../test.csv' out_dict = { 'Class1': ['A', 'B'], 'Class2': ['C', 'D', 'E', 'F', 'G', 'H', 'I'], 'Class3': ['J', 'K', 'L', 'M', 'N']} # write dictionary to csv with open(path, 'wt', newline='') as csv_file: writer = csv.writer(csv_file) writer.writerow(out_dict.keys()) writer.writerows(itertools.zip_longest(*out_dict.values())) csv_file.close() # read csv as dictionary with open(path, 'rt') as csv_file: reader = csv.reader(csv_file); in_dict = ??? csv_file.close() print(in_dict)  Desired Output: {'Class1': ['A', 'B'], 'Class2': ['C', 'D', 'E', 'F', 'G', 'H', 'I'], 'Class3': ['J', 'K', 'L', 'M', 'N']} 

Answer: To read the CSV file back I recommend to use csv.DictReader: import csv import itertools path = '<PATH>' out_dict = { "Class1": ["A", "B"], "Class2": ["C", "D", "E", "F", "G", "H", "I"], "Class3": ["J", "K", "L", "M", "N"], } # write dictionary to csv with open(path, 'wt', newline='') as csv_file: writer = csv.writer(csv_file) writer.writerow(out_dict.keys()) writer.writerows(itertools.zip_longest(*out_dict.values())) # read csv as dictionary out = {} with open(path, 'rt') as csv_file: reader = csv.DictReader(csv_file) for row in reader: for k, v in row.items(): if v != '': out.setdefault(k, []).append(v) print(out)  Prints: { "Class1": ["A", "B"], "Class2": ["C", "D", "E", "F", "G", "H", "I"], "Class3": ["J", "K", "L", "M", "N"], }
Question: I would like to replace everything in String untill "err" or "error" appear for empty string. So: "abc err efg" -> "efg", "abc error efg" -> "efg. How to do it with one pattern using re.sub? I tried this: lines_input = ['some line err hello', 'some line error hello'] rep = {r'^.*?err': '', r'^.*?error': ''} dict((re.escape(k), v) for k, v in rep.items()) pattern = re.compile("|".join(rep.keys())) for line in lines_input: print pattern.sub(lambda m: rep[re.escape(m.group(0))], line)  and had KeyError. Would like to have: hello hello 

Answer: You don't need multiple regular expressions. The two patterns are the same except for the optional or at the end of error, so use an optional group. line = re.sub(r'.*err(or)?\s*', '', line)