eshangj/stackoverflow_q_and_a_sample huggingface dataset:
Question: What I need: I have a dataframe where the elements of a column are lists. There are no duplications of elements in a list. For example, a dataframe like the following: import pandas as pd >>d = {'col1': [[1, 2, 4, 8], [15, 16, 17], [18, 3], [2, 19], [10, 4]]} >>df = pd.DataFrame(data=d) col1 0 [1, 2, 4, 8] 1 [15, 16, 17] 2 [18, 3] 3 [2, 19] 4 [10, 4]  I would like to obtain a dataframe where, if at least a number contained in a list at row i is also contained in a list at row j, then the two list are merged (without duplication). But the values could also be shared by more than two lists, in that case I want all lists that share at least a value to be merged.  col1 0 [1, 2, 4, 8, 19, 10] 1 [15, 16, 17] 2 [18, 3]  The order of the rows of the output dataframe, nor the values inside a list is important.  What I tried: I have found this answer, that shows how to tell if at least one item in list is contained in another list, e.g. >>not set([1, 2, 4, 8]).isdisjoint([2, 19]) True  Returns True, since 2 is contained in both lists. I have also found this useful answer that shows how to compare each row of a dataframe with each other. The answer applies a custom function to each row of the dataframe using a lambda. df.apply(lambda row: func(row['col1']), axis=1)  However I'm not sure how to put this two things together, how to create the func method. Also I don't know if this approach is even feasible since the resulting rows will probably be less than the ones of the original dataframe. Thanks!

Answer: This is not straightforward. Merging lists has many pitfalls. One solid approach is to use a specialized library, for example networkx to use a graph approach. You can generate successive edges and find the connected components. Here is your graph:  You can thus:  generate successive edges with add_edges_from find the connected_components craft a dictionary and map the first item of each list groupby and merge the lists (you could use the connected components directly but I'm giving a pandas solution in case you have more columns to handle)  import networkx as nx G = nx.Graph() for l in df['col1']: G.add_edges_from(zip(l, l[1:])) groups = {k:v for v,l in enumerate(nx.connected_components(G)) for k in l} # {1: 0, 2: 0, 4: 0, 8: 0, 10: 0, 19: 0, 16: 1, 17: 1, 15: 1, 18: 2, 3: 2} out = (df.groupby(df['col1'].str[0].map(groups), as_index=False) .agg(lambda x: sorted(set().union(*x))) )  output:  col1 0 [1, 2, 4, 8, 10, 19] 1 [15, 16, 17] 2 [3, 18]
Question: I have the dataframe below and want to filter by time. The time column comes up as an object when I use dtypes. To get the time to use as the filter criteria I use split: start_time = "25 September 2022, 13:00:00" split_time = start_time.split(", ")[1]  I have tried converting split_time and the df column to datime but get an error on the df column conversion: TypeError: <class 'datetime.time'> is not convertible to datetime I have also tried a simple string search but this doesn't return any results. I have been able to filter by date using: split_date = start_time.split(", ")[0] event_date = datetime.strptime(split_date, "%d %B %Y") events_df['start_date'] = pd.to_datetime(events_df['start_date']) filtered_df = events_df.loc[(events_df['start_date'] == event_date)]  But can't seem to do the equivalent for time. Can anyone see the problem? Thanks      fixture_id name start_date time     145 9394134 Plymouth Argyle v Ipswich Town 2022-09-25 00:00:00 12:30:00   146 9694948 Grays Athletic v Merstham FC 2022-09-25 00:00:00 13:00:00   147 9694959 FC Romania v Faversham Town 2022-09-25 00:00:00 15:00:00    

Answer: Comapre times generated by Series.dt.time with Timestamp.time: start_time = "25 September 2022, 13:00:00" dt = pd.to_datetime(start_time, format="%d %B %Y, %H:%M:%S") events_df['start_date'] = pd.to_datetime(events_df['start_date']) #if necessary events_df['time'] = pd.to_datetime(events_df['time']).dt.time filtered_df = events_df.loc[(events_df['time'] == dt.time())] print (filtered_df) fixture_id name start_date time 1 146 9694948 Grays Athletic v Merstham FC 2022-09-25 13:00:00
Question: I want to plot this dataframe like a time series, a line for every country that every year increases or decreases according to 'count'. How can i do this?  country count Year 2005 Australia 2 2005 Austria 1 2005 Belgium 0 2005 Canada 4 2005 China 0 2006 Australia 3 2006 Austria 0 2006 Belgium 1 2006 Canada 5 2006 China 2 2007 Australia 5 2007 Austria 1 2007 Belgium 2 2007 Canada 6 2007 China 3  I'd like a thing like this: 

Answer: You can use seaborn.lineplot: import seaborn as sns df.Year = pd.to_datetime(df.Year) sns.set(rc={'figure.figsize':(12, 8)}) # changed the figure size to avoid overlapping sns.lineplot(data=df, x=df['Year'].dt.strftime('%Y'), # show only years with strftime y=df['count'], hue='country')  which gives
Question: I wrote a code using the pyproj library and converted this code to an exe file for use on another computer. I added the pyproj to the requirements.txt file. And I installed the library with the requirements.txt file on the other computer I will use. When I run the exe file, I get the following error: importlib.metadata.PackageNotFoundError: No package metadata was found for pyproj  I'd be glad if you can help.

Answer: Here is my solution for those who have problems like above while running the exe file:  pyinstaller --onefile --copy-metadata pyproj "example.py"  For now this seems to have fixed the problem.
Question: I'm trying to use pd and sqlalchemy to run all the sql files in a directory. Currently I can load the text of a sql file into a sqlalchemy.sql.text object, and send that directly to pd.read_sql. What should I use to locate the bind parameters within my sql script so that I can prompt the user for them? import sys, os, pandas as pd, re, sqlalchemy as sa os.chdir(sys.argv[1]) os.mkdir("out") uname = input("Username >>> ") passw = input("Password >>> ") engine = sa.create_engine(f"oracle+cx_oracle://{uname}:{passw}@PROD/?encoding=UTF-8&nencoding=UTF-8") for filename in os.listdir('.'): if not re.match(filename,r".*\.sql"): continue #Ignore non-sql files. print("Executing",filename) with open(filename,"r") as my_file: sql = sa.text(''.join(filename.readlines())) ### Something goes here ### df = pd.read_sql(''.join(my_file.readlines()),engine) df.to_excel(f"./out/{filename}",filename,index=False)  My current best guess is to read through the file line-by-line with a regex that finds things that look like bind params, but I feel like there should be a better way.

Answer: You can obtain the names of the bind parameters from the compiled query: >>> q = text('select id, name from users where name = :p1 and age > :p2') >>> q.compile().params {'p1': None, 'p2': None}  In the resulting dictionary, the keys correspond to the bind parameter names. The values are None until values are bound.
Question: Please see this -  I want to create a table similar to this using pandas in python? Can anyone guide? Unable to understand how to add "Total Marks" text corresponding to two columns ("Student id" & "Course id"). Thanks

Answer: As far as I'm aware, pandas doesn't really support merged cells/cells spanning multiple rows in the same way as Excel/HTML do. A pandas-esque way to include the total row for a single column would be something like the following: import pandas as pd columns = ['Student ID', 'Course ID', 'Marks'] data = [(103, 201, 67), (103, 203, 67), (103, 204, 89)] df = pd.DataFrame(data, columns=columns) df.at['Total marks', 'Marks'] = df['Marks'].sum() print(df)  (per this answer) This gives you:  Student ID Course ID Marks 0 103.0 201.0 67.0 1 103.0 203.0 67.0 2 103.0 204.0 89.0 Total marks NaN NaN 223.0  For HTML output like the image you provided, it looks like you would have to edit the HTML after exporting from pandas (this thread talks about columns but I believe the same applies for merging cells in rows).
Question: Given a number n, count number of occurrences of digits 0, 2 and 4 including n. Example1: n = 10 output: 4   Example2: n = 22 output: 11   My Code: n = 22 def count_digit(n): count = 0 for i in range(n+1): if '2' in str(i): count += 1 if '0' in str(i): count += 1 if '4' in str(i): count += 1 return count count_digit(n)  Code Output: 10 Desired Output: 11 Constraints: 1 <= N <= 10^5 Note: The solution should not cause outOfMemoryException or Time Limit Exceeded for large numbers.

Answer: Another brute force, seems faster: def count_digit(n): s = str(list(range(n+1))) return sum(map(s.count, '024'))  Benchmark with n = 10**5: result time solution 115474 244 ms original 138895 51 ms Kelly 138895 225 ms islam_abdelmoumen 138895 356 ms CodingDaveS  Code (Try it online!): from timeit import default_timer as time def original(n): count = 0 for i in range(n+1): if '2' in str(i): count += 1 if '0' in str(i): count += 1 if '4' in str(i): count += 1 return count def Kelly(n): s = str(list(range(n+1))) return sum(map(s.count, '024')) def islam_abdelmoumen(n): count = 0 for i in map(str,range(n+1)): count+=i.count('0') count+=i.count('2') count+=i.count('3') return count def CodingDaveS(n): count = 0 for i in range(n + 1): if '2' in str(i): count += str(i).count('2') if '0' in str(i): count += str(i).count('0') if '4' in str(i): count += str(i).count('4') return count funcs = original, Kelly, islam_abdelmoumen, CodingDaveS print('result time solution') print() for _ in range(3): for f in funcs: t = time() print(f(10**5), ' %3d ms ' % ((time()-t)*1e3), f.__name__) print()
Question: import re re_for_identificate_1 = r"" with open("data_path/filename_1.txt","r+") as file: for line in file: #replace with a substring adding a space in the middle line = re.sub(re_for_identificate_1, " milesimo", line) #replace in txt with the fixed line  Example filename_1.txt : unmilesimo primero 1001° dosmilesimos quinto 2005° tresmilesimos 3000° nuevemilesimos doceavo 9012°  The correct output file that I need obtiene is this: Rewrited input filename_1.txt un milesimo primero 1001° dos milesimos quinto 2005° tres milesimos 3000° nueve milesimos doceavo 9012°  What is the regex that I need and what is the best way to replace the fixed línes in their original positions in the input file?

Answer: You can use file.seek(0) to go beginning of the file, then write data and truncate the file. Like this: import re re_for_identificate_1 = "(?<!^)milesimo" tmp = "" with open("data.txt", "r+") as file: for line in file: line = re.sub(re_for_identificate_1, " milesimo", line) tmp += line file.seek(0) file.write(tmp) file.truncate()  The regex you want to use is "(?<!^)milesimo" to replace every instance of "milesimo" with " milesimo" but not at the beginning of a line.
Question: I will give the specific case and the generic case so we can help more people: I have a list of ordered lists and another list with the same length as each ordered list. Each list in the list of lists is students' answers from a large-scale evaluation, and the second is the correct answers from the test. I need to check the % of right answers, AKA how many matches are between each item in each of the lists in order. The output should be a list where 1 means there is a match, and 0 there is no match. Example: list1 = [['A', 'B', 'C', 'A'], ['A', 'C', 'C', 'B']] list2 = ['A', 'B', 'C', 'A'] result = [[1, 1, 1, 1],[1, 0, 1, 0]  Thank you!

Answer: The others have done a fine job detailing how this can be done with list comprehensions. The following code is a beginner-friendly way of getting the same answer. final = [] # Begin looping through each list within list1 for test in list1: # Create a list to store each students scores within student = [] for student_ans, correct_ans in zip(test, list2): # if student_ans == correct_ans at the same index... if student_ans == correct_ans: student.append(1) else: student.append(0) # Append the student list to the final list final.append(student)
Question: This is a question from someone who tries to answer questions about pandas dataframes. Consider a question with a given dataset which is just the visualization (not the actual code), for example:  numbers letters dates all 0 1 a 20-10-2020 NaN 1 2 b 21-10-2020 b 2 3 c 20-11-2020 4 3 4 d 20-10-2021 20-10-2020 4 5 e 10-10-2020 3.14  Is it possible to quickly import this in python as a dataframe or as a dictionary? So far I copied the given text and transformed it to a dataframe by making strings (adding '') and so on. I think there are two 'solutions' for this:  Make a function that given the text as input, it somehow transforms it to a dataframe. Use some function in the text-editor (I use spyder) which can do this trick for us. 

Answer: read_clipboard You can use pd.read_clipboard() optionally with a separator (e.g. pd.read_clipboard('\s\s+') if you have datetime strings or spaces in column names and columns are separated by at least two spaces):  select text on the question and copy to clipboard (ctrl+c/command-c) move to python shell or notebook and run pd.read_clipboard()  Note that this doesn't work well on all platforms. read_csv + io.StringIO For more complex formats, combine read_csv combined with io.StringIO: data = ''' numbers letters dates all 0 1 a 20-10-2020 NaN 1 2 b 21-10-2020 b 2 3 c 20-11-2020 4 3 4 d 20-10-2021 20-10-2020 4 5 e 10-10-2020 3.14 ''' import io df = pd.read_csv(io.StringIO(data), sep='\s+') df
Question: Here is the Python statement creating the problem: nan_rows = df[df.isnull().any(1)]  And it gives the following warning: FutureWarning: In a future version of pandas all arguments of DataFrame.any and Series.any will be keyword-only.  It is a warning from production environment. I am sorry that I am not able to share the complete code as it is fairly complex to fit into this question. How can I understand what this warning is saying, and how should I update this line of code?

Answer: Starting from pandas 1.5, you get a FutureWarning. You must specify axis=1: nan_rows = df[df.isnull().any(axis=1)]  In a future version only keywords arguments will be accepted, omitting the keyword will raise an error.
Question: So I have this data that list of True and False for example tf = ['True', 'False', 'False']  how can I convert tf to a bool. Once I print(tf[0]) it prints True 

Answer: Use the ast module: import ast tf = ['True', 'False', 'False'] print(type(ast.literal_eval(tf[0]))) print(ast.literal_eval(tf[0]))  Result: <class 'bool'> True  Ast Documentation Literal_eval
Question: I am trying out a simple text-matching activity where I scraped titles of blog posts and try to match it with my pre-defined categories once I find specific keywords. So for example, the title of the blog post is   "Capture Perfect Night Shots with the Oppo Reno8 Series" Once I ensure that "Oppo" is included in my categories, "Oppo" should match with my "phone" category like so: categories = {"phone" : ['apple', 'oppo', 'xiaomi', 'samsung', 'huawei', 'nokia'], "postpaid" : ['signature', 'postpaid'], "prepaid" : ['power all', 'giga'], "sku" : ['data', 'smart bro'], "ewallet" : ['gigapay'], "event" : ['gigafest'], "software" : ['ios', 'android', 'macos', 'windows'], "subculture" : ['anime', 'korean', 'kpop', 'gaming', 'pop', 'culture', 'lgbtq', 'binge', 'netflix', 'games', 'ml', 'apple music'], "health" : ['workout', 'workouts', 'exercise', 'exercises'], "crypto" : ['axie', 'bitcoin', 'coin', 'crypto', 'cryptocurrency', 'nft'], "virtual" : ['metaverse', 'virtual']}  Then my dataframe would look like this Fortunately I found a reference to how to use regex in mapping to nested dictionaries but it can't seem to work past the first couple of words Reference is here So once I use the code def put_category(cats, text): regex = re.compile("(%s)" % "|".join(map(re.escape, categories.keys()))) if regex.search(text): ret = regex.search(text) return ret[0] else: return 'general'  It usually reverts to put "general" as the category, even when doing it in lowercase as seen here I'd prefer to use the current method of inputting values inside the dictionary for this matching activity instead of running pure regex patterns and then putting it through fuzzy matching for the result.

Answer: You can create a reverse mapping that maps keywords to categories instead, so that you can efficiently return the corresponding category when a match is found: mapping = {keyword: category for category, keywords in categories.items() for keyword in keywords} def put_category(mapping, text): match = re.search(rf'\b(?:{"|".join(map(re.escape, mapping))})\b', text, re.I) if match: return mapping[match[0].lower()] return 'general' print(put_category(mapping, "Capture Perfect Night Shots with the Oppo Reno8 Series"))  This outputs: phone  Demo: https://replit.com/@blhsing/BlandAdoredParser
Question: I have NLTK installed and it is giving me an error: Resource punkt not found. Please use the NLTK Downloader to obtain the resource: import nltk nltk.download('punkt') For more information see: https://www.nltk.org/data.html Attempted to load tokenizers/punkt/PY3/english.pickle Searched in: - '/Users/divyanshundley/nltk_data' - '/Library/Frameworks/Python.framework/Versions/3.10/nltk_data' - '/Library/Frameworks/Python.framework/Versions/3.10/share/nltk_data' - '/Library/Frameworks/Python.framework/Versions/3.10/lib/nltk_data' - '/usr/share/nltk_data' - '/usr/local/share/nltk_data' - '/usr/lib/nltk_data' - '/usr/local/lib/nltk_data' - '' **********************************************************************  And my code is: import nltk nltk.download('punkt') def tokenize(token): return nltk.word_tokenize(token); tokenize("why is this not working?"); 

Answer: Adding these code will resolve your issue: nltk.download('punkt') nltk.download('wordnet') nltk.download('omw-1.4')
Question: I want to apply aggregation functions like sum, mean, etc to a column of type List[f64] after a group_by such that I get a List[64] entry back.  Say I have: import polars as pl df = pl.DataFrame( { "Case": ["case1", "case1"], "List": [[1, 2, 3], [4, 5, 6]], } ) print(df)  shape: (2, 2) ┌───────┬────────────┐ │ Case ┆ List │ │ --- ┆ --- │ │ str ┆ list[i64] │ ╞═══════╪════════════╡ │ case1 ┆ [1, 2, 3] │ │ case1 ┆ [4, 5, 6] │ └───────┴────────────┘  I want to group_by Case and sum List so that I end up with: ┌───────┬────────────┐ │ Case ┆ List │ │ --- ┆ --- │ │ str ┆ list[i64] │ ╞═══════╪════════════╡ │ case1 ┆ [5, 7, 9] │ └───────┴────────────┘  How would I best do this? Note that the length of each of the lists are 256, so indexing each of them is not a good solution. Thanks!

Answer:  Note that the length of each of the lists are 256, so indexing each of them is not a good solution.  If you are sure of the length of your lists ahead of time, then we can avoid the typical explode/index/group_by solution as follows: list_size = 3 ( df.group_by("Case") .agg( pl.concat_list( pl.col("List") .list.slice(n, 1) .list.first() .sum() for n in range(0, list_size) ) ) )  shape: (1, 2) ┌───────┬───────────┐ │ Case ┆ List │ │ --- ┆ --- │ │ str ┆ list[i64] │ ╞═══════╪═══════════╡ │ case1 ┆ [5, 7, 9] │ └───────┴───────────┘  How it works To see how this works, let's look at how the algorithm adds the first elements of each list. (We can extrapolate for all elements from this example.) In the first step, we use group_by to accumulate all the lists for each Case. ( df.group_by("Case") .agg( pl.concat_list("List") ) )  shape: (1, 2) ┌───────┬────────────────────────┐ │ Case ┆ List │ │ --- ┆ --- │ │ str ┆ list[list[i64]] │ ╞═══════╪════════════════════════╡ │ case1 ┆ [[1, 2, 3], [4, 5, 6]] │ └───────┴────────────────────────┘  The next step is to slice each list so that we get only the nth element of each list. In this example, we want only the first element of each list, corresponding to the 0 in slice(0, 1). Notice that the internal lists are now all only one element each. ( df.group_by("Case") .agg( pl.concat_list( pl.col("List").list.slice(0, 1) ) ) )  shape: (1, 2) ┌───────┬─────────────────┐ │ Case ┆ List │ │ --- ┆ --- │ │ str ┆ list[list[i64]] │ ╞═══════╪═════════════════╡ │ case1 ┆ [[1], [4]] │ └───────┴─────────────────┘  In the last step, we sum the individual elements: ( df.groupby("Case") .agg( pl.concat_list( pl.col("List") .list.slice(0, 1) .list.first() .sum() ) ) )  shape: (1, 2) ┌───────┬───────────┐ │ Case ┆ List │ │ --- ┆ --- │ │ str ┆ list[i64] │ ╞═══════╪═══════════╡ │ case1 ┆ [5] │ └───────┴───────────┘  To accomplish this for all n elements of each list, we simply write our expressions using a list comprehension, substituting n in our slice(n, 1).
Question: I have a Polars dataframe with a column of type str with the date and time df = pl.from_repr(""" ┌─────────────────────────┐ │ EventTime │ │ --- │ │ str │ ╞═════════════════════════╡ │ 2020-03-02T13:10:42.550 │ └─────────────────────────┘ """)  I want to convert this column to the polars.Datetime type. After reading this post Easily convert string column to pl.datetime in Polars, I came up with: df = df.with_columns(pl.col('EventTime').str.to_datetime("%Y-%m-%dT%H:%M:%f", strict=False))  However, the values my column "EventTime' are all null. Many Thanks!

Answer: You were close. You forgot the seconds component of your format specifier: ( df .with_columns( pl.col('EventTime') .str.to_datetime( format="%Y-%m-%dT%H:%M:%S%.f", strict=False) .alias('parsed EventTime') ) )  shape: (1, 2) ┌─────────────────────────┬─────────────────────────┐ │ EventTime ┆ parsed EventTime │ │ --- ┆ --- │ │ str ┆ datetime[ns] │ ╞═════════════════════════╪═════════════════════════╡ │ 2020-03-02T13:10:42.550 ┆ 2020-03-02 13:10:42.550 │ └─────────────────────────┴─────────────────────────┘  BTW, the format you are using is standard, so you can eliminate the format specifier altogether. ( df .with_columns( pl.col('EventTime') .str.to_datetime() .alias('parsed EventTime') ) )  shape: (1, 2) ┌─────────────────────────┬─────────────────────────┐ │ EventTime ┆ parsed EventTime │ │ --- ┆ --- │ │ str ┆ datetime[μs] │ ╞═════════════════════════╪═════════════════════════╡ │ 2020-03-02T13:10:42.550 ┆ 2020-03-02 13:10:42.550 │ └─────────────────────────┴─────────────────────────┘  Edit  And what if I would like to ignore the miliseconds? so the "%.f", if I just leave it out it can't interpret properly the dataframe  We need to allow Polars to parse the date string according to the actual format of the string. That said, after the parsing, we can use dt.truncate to throw away the fractional part. ( df .with_columns( pl.col('EventTime') .str.to_datetime() .dt.truncate('1s') .alias('parsed EventTime') ) )  shape: (1, 2) ┌─────────────────────────┬─────────────────────┐ │ EventTime ┆ parsed EventTime │ │ --- ┆ --- │ │ str ┆ datetime[μs] │ ╞═════════════════════════╪═════════════════════╡ │ 2020-03-02T13:10:42.550 ┆ 2020-03-02 13:10:42 │ └─────────────────────────┴─────────────────────┘
Question: I would like to use a different logging level in development and production. To do so, I need early in my program to set the minimal level for logs to be triggered. The default is to output all severities: from loguru import logger as log log.debug("a debug log") log.error("an error log") # output # 2022-09-15 16:51:23.325 | DEBUG | __main__:<module>:3 - a debug log # 2022-09-15 16:51:23.327 | ERROR | __main__:<module>:4 - an error log  There is a Changing the level of an existing handler section in the documentation, that states among others that  Once a handler has been added, it is actually not possible to update it. (...) The most straightforward workaround is to remove() your handler and then re-add() it with the updated level parameter.  My problem is that I have not added anything, so there is nothing to remove. I also cannot modify log. So what should I do?

Answer: Like so: import sys from loguru import logger as log log.remove() #remove the old handler. Else, the old one will work along with the new one you've added below' log.add(sys.stderr, level="INFO") log.debug("debug message") log.info("info message") log.warning("warning message") log.error("error message")  You can use sys.stdout or other file object, in place of sys.stderr. More info in this related GitHub thread.