eshangj/stackoverflow_q_and_a_sample huggingface dataset:
Question: I am trying to append > 1000 csv files (all with the same format) in Python. The files have anything between 1KB and 30GB. All the files have 200GB in total. I want to combine these files into a unique dataframe. Below what I am doing which is very very slow: folder_path = 'Some path here' csv_files = [file for file in os.listdir(folder_path) if file.endswith('.csv')] combined_data = pd.DataFrame() for e, csv_file in enumerate(csv_files): print(f'Processing {e+1} out of {len(csv_files)}: {csv_file}') combined_data = pd.concat([combined_data, pd.read_csv(os.path.join(folder_path, csv_file), dtype={'stringbvariablename': str})])  One of the variables is a string. Everything else numbers. RAM memory is not an issue since I am using a cluster.

Answer: If all of your files are in the same format, and you are just trying to create a new CSV, then ditch pandas. import csv import pathlib def write_to_writer(writer: csv.writer, path: pathlib.Path, skipheader=False) -> None: with path.open(newline="") as f: reader = csv.reader(f) if skipheader: next(reader) writer.writerows(reader) folder_path = pathlib.Path('Some path here') csv_files = [p for p in folder_path.iterdir() if p.suffix =='.csv'] with open('combined_data.csv', 'w', newline="") as fout: writer = csv.writer(fout) # handle first csv, with header first_path, *rest = csv_paths write_to_writer(writer, first_path, skipheader=False) for path in rest: write_to_writer(writer, path, skipheader=True)  This avoids the pd.concat in a loop problem, which is a classic antipattern. However, if you need a large dataframe (because you are going to actually use the dataframe), then your problem is fixed simply by appending to a list and then concating at the end: import pathlib import pandas as pd folder_path = pathlib.Path('Some path here') csv_paths = [p for p in folder_path.iterdir() if p.suffix =='.csv'] df = pd.concat([ pd.read_csv(path, dtype={'stringbvariablename': str}) for path in csv_paths ])  Again, this avoids the anti-pattern (because .appending to a list in a loop is efficient)
Question: I have string data that I am matching to a list of terms and I want to create a new column that shows all words found in each row of the dataframe df = pd.DataFrame({'String': ['Cat Dog Fish', 'Cat Dog', 'Pig Horse', 'DogFish']}) print(df) String 0 Cat Dog Fish 1 Cat Dog 2 Pig Horse 3 DogFish 4 CatHorse words = ['Cat', 'Dog', 'Fish']  I know using df.loc[df['String'].str.contains('|'.join(words))].copy() will return all rows that contain at least one of the terms I'm searching for but I would like output that keeps a record of which terms are found in each string like this:  String Matched 0 Cat Dog Fish [Cat, Dog, Fish] 1 Cat Dog [Cat, Dog] 2 DogFish [Dog, Fish] 3 CatHorse [Cat]  or even just  String Matched 0 Cat Dog Fish CatDogFish 1 Cat Dog CatDog 2 DogFish DogFish 3 CatHorse Cat  Not sure where to begin with making this column, any help is appreciated

Answer: Try: words = ["Cat", "Dog", "Fish"] df["Matched"] = df["String"].apply(lambda s: [w for w in words if w in s]) print(df)  Prints:  String Matched 0 Cat Dog Fish [Cat, Dog, Fish] 1 Cat Dog [Cat, Dog] 2 Pig Horse [] 3 DogFish [Dog, Fish]
Question: I have a DataFrame with key and value columns. value is sometimes NA: df = pd.DataFrame({ 'key': np.random.randint(0, 1_000_000, 100_000_000), 'value': np.random.randint(0, 1_000, 100_000_000).astype(float), }) df.loc[df.value == 0, 'value'] = np.nan  I want to group by key and sum over the value column. If any value is NA for a key, I want the sum to be NA. The code in this answer takes 35.7 seconds on my machine: df.groupby('key')['value'].apply(np.array).apply(np.sum)  This is a lot slower than what is theoretically possible. The built-in Pandas SeriesGroupBy.sum takes 6.31 seconds on my machine: df.groupby('key')['value'].sum()  but it doesn't support NA handling (see this GitHub issue). What code can I write to get comparable performance to the built-in operator while still handling NaNs?

Answer: One workaround could be to replace the NaNs by Inf: df.fillna({'value': np.inf}).groupby('key')['value'].sum().replace(np.inf, np.nan)  Faster alternative: df['value'].fillna(np.inf).groupby(df['key']).sum().replace(np.inf, np.nan)  Example output: key 0 45208.0 1 NaN 2 62754.0 3 50001.0 4 51073.0 ... 99995 55102.0 99996 43048.0 99997 49497.0 99998 43301.0 99999 NaN Name: value, Length: 100000, dtype: float64  Timing (on 10m rows). # original sum 743 ms ± 81 ms per loop (mean ± std. dev. of 7 runs, 1 loop each) # Inf workaround 918 ms ± 70.6 ms per loop (mean ± std. dev. of 7 runs, 1 loop each) # Inf workaround (alternative) 773 ms ± 60.6 ms per loop (mean ± std. dev. of 7 runs, 1 loop each) # custom apply with numpy 5.99 s ± 263 ms per loop (mean ± std. dev. of 7 runs, 1 loop each)
Question: Is there any way to pass parameters to a Python function such that if the variable name matches the parameter name, the function will ensure the input is correct? Let's say I have a function as such: def func(foo, bar): return do_something_with(foo, bar)  and I wanted to call it as such: foo = get_foo() bar = get_bar() ret_val = func(foo=foo, bar=bar)  I can do this, but I feel like when the parameter name and the variable name are the same, there should be some mechanism to automatically match up the parameters (like NATURAL JOIN in SQL). Is there any such mechanism?

Answer: There is no special function call syntax to pass variables as keyword arguments of the same name. If that's how you want to call your function, you must write func(foo=foo, bar=bar). While there are a few ways you could work around the issue, they're all so much worse than just writing the keywords out that I wouldn't recommend them. As an example of a bad workaround, you could define your function to take arbitrary extra keyword arguments (with def func(foo, bar, **kwargs_to_be_ignored)) and then call the function with func(**locals()). That will pass in every local variable in your namespace as a keyword argument. In the body of some functions with few local variables, this might be reasonably clean, but in many other contexts (especially at the global level) it will very often pass a whole mess of unnecessary stuff. And if your function has optional arguments (e.g. def func(foo, bar, baz='default)) you can wind up accidentally passing in an overriding value to an argument you didn't want to pass if you just happen to have a variable with that argument's name in your local namespace.
Question: I have been spending time on this regex but I can't get it to work. So I need to match bunch of words in a phrase but if the same word occurs with a set of words, I do not want that to be captured. For example: phrase: Hi, I am talking about a recall on the product I bought last month. If I recall correctly, I purchased this at your store on august 15th. Can you tell me if I can get a refund on this recall? Result should match the first recall and the last recall. but it should not match 'If I recall' since those three words together doesn't talk about the product recall. I tried different variations of this but couldn't get it to work. This matches all 'recall' terms. (?<!If\sI\srecall).*?(recalls?|recalled).*?(?!If\sI\srecall)  I am using Python 3.10 to test this. Any help would be appreciated.

Answer: If you want to match the 2 words: (?<!\bIf\sI\s)\brecall(?:s|ed)?\b  The pattern matches:  (?<!\bIf\sI\s) Negative lookbehind, assert not If I  to the left \brecall(?:s|ed)? Match one of the words recall recalls recalled \b A word boundary  Regex demo | Python demo
Question: I have a Python environment with Django 3 that has to use GDAL (because the DB engine I use is django.contrib.gis.db.backends.postgis). After an upgrade of GDAL (from 3.6.4_6 to 3.7.1_1), I have this exception on any command to run the django project :  File "~/.pyenv/versions/3.8.9/lib/python3.8/ctypes/__init__.py", line 373, in __init__ self._handle = _dlopen(self._name, mode) OSError: dlopen(/usr/local/lib/libgdal.dylib, 0x0006): Symbol not found: __ZN3Aws6Client19ClientConfigurationC1Ev Referenced from: <FA8C3295-2793-3C69-A419-16C41753696B> /opt/homebrew/Cellar/apache-arrow/12.0.1_4/lib/libarrow.1200.1.0.dylib Expected in: <BDB1F1E3-0BE9-3D7D-A57E-9D9F8CAD197A> /opt/homebrew/Cellar/aws-sdk-cpp/1.11.145/lib/libaws-cpp-sdk-core.dylib  I've managed to isolate the problem in a fresh python environment with only loading the GDAL library : from ctypes import CDLL; CDLL("/usr/local/lib/libgdal.dylib"). Note, I'm on a Mac (CPU M2 Pro), I've installed GDAL and all its dependancies via brew. I've tried reinstalling it from fresh but it didn't change a thing and I've also tried with different python versions. Aftermatch: An issue was opened soon after my post (#140082) and the pull request (!140127) fixing it was merged on the 22nd of August 2023. So you just need to do brew update && brew upgrade

Answer: I faced the same OSError: dlopen(/usr/local/lib/libgdal.dylib, 0x0006): Symbol not found: __ZN3Aws6Client19ClientConfigurationC1Ev . Applied the approach suggested by @bennylope (https://nelson.cloud/how-to-install-older-versions-of-homebrew-packages/) and got it worked: wget https://raw.githubusercontent.com/Homebrew/homebrew-core/824c551b4f514e7cd86358133924906b982c87e2/Formula/postgis.rb wget https://raw.githubusercontent.com/Homebrew/homebrew-core/0faa586d8415cd2ced33345aa3e13c664c92aeda/Formula/a/aws-sdk-cpp.rb brew install postgis.rb brew uninstall --ignore-dependencies aws-sdk-cpp brew install aws-sdk-cpp.rb brew link aws-sdk-cpp
Question: I running the following code in python 3.10.12. There is a sync function fetch_data, which should return 1 immediately. And in main function, a CPU-bound task like sum(i for i in range(int(1e6))) which cost 4~5s. import asyncio def fetch_data(): print("done") return 1 async def main(): loop = asyncio.get_event_loop() task = loop.run_in_executor(None, fetch_data) sum(i for i in range(int(1e6))) print(task.done()) asyncio.run(main())  The output I expected is done True  But what I got is done False  The fetch_data seems never done, but when I insert await asyncio.sleep(0) before print(task.done()), the output is what I expected. import asyncio def fetch_data(): print("done") return 1 async def main(): loop = asyncio.get_event_loop() task = loop.run_in_executor(None, fetch_data) sum(i for i in range(int(1e6))) await asyncio.sleep(0) # After sleep(0), the output is as expected. print(task.done()) asyncio.run(main())  Why is this happening? What should I do to achieve what I needed without asyncio.sleep(0)?

Answer: It's happening because you don't give the executor a change to process it's tasks. With await asyncio.sleep(0) you interrupt your main() function and executor process the tasks and the result is as expected. If asyncio.sleep is problem, you can use other executor: import asyncio import concurrent.futures def fetch_data(): print("done", flush=True) print() return 1 async def main(): with concurrent.futures.ThreadPoolExecutor(max_workers=2) as executor: task = executor.submit(fetch_data) sum(i for i in range(int(1e8))) print(task.done()) asyncio.run(main())  Prints: done True
Question: I want to obtain a list of all the unique characters in a text. A particularity of the text is that it includes composed characters like s̈, b̃. So when I split the text, the special characters are separated. For example, this character s̈ is separated into two characters s and ¨. This is an example of the text I want to process. sentence = "nejon ámas̈hó T̃iqu c̈ab̃op" print(sentence) print(list[set(sentence)])  I want to obtain a list with the unique characters. For this sentence, this list should be expected_list = ['a', 'á', 'b̃', 'c̈', 'e', 'h', 'i', 'j', 'm', 'n', 'o', 'ó', 'p', 'q', 's̈', 'T̃', 'u' ] but it is actual_list = ['j', 'p', 'c', 'n', 'a', ' ', 'i', 'á', 'o', 'T', 'u', '̃', 'h', '̈', 'q', 's', 'e', 'm', 'b', 'ó'] I was reading that I can normalize the special characters as follows import unicodedata # Only for the character s̈ print(ascii(unicodedata.normalize('NFC', '\u0073\u00a8'))) #prints 's\xa8'  But I don't know how to continue. Any help would be greatly appreciated.

Answer: Handling composed characters in Python can be a bit tricky due to the nature of how they are encoded. Try the grapheme library, which specifically deals with grapheme clusters (textual units that are displayed as a single character) Install the grapheme library using pip: pip install grapheme  or I prefer this way (to make sure it's installing to the current python binary dirs) python3 -m pip install grapheme  Then, you can use it to extract the unique grapheme clusters from the sentence: import grapheme sentence = "nejon ámas̈hó T̃iqu c̈ab̃op" unique_characters = list(grapheme.graphemes(sentence)) print(unique_characters)
Question: I'm currently working on a project where I need to perform real-time sentiment analysis on live audio streams using Python. The goal is to analyze the sentiment expressed in the spoken words and provide insights in real-time. I've done some research and found resources on text-based sentiment analysis, but I'm unsure about how to adapt these techniques to audio streams. Context and Efforts: Research: I've researched various libraries and tools for sentiment analysis, such as Natural Language Processing (NLP) libraries like NLTK and spaCy. However, most resources I found focus on text data rather than audio. Audio Processing: I'm familiar with libraries like pyaudio and soundfile in Python for audio recording and processing. I've successfully captured live audio streams using these libraries. Text-to-Speech Conversion: I've experimented with converting the spoken words from the audio streams into text using libraries like SpeechRecognition to prepare the data for sentiment analysis. Challenges: Sentiment Analysis: My main challenge is adapting the sentiment analysis techniques to audio data. I'm not sure if traditional text-based sentiment analysis models can be directly applied to audio, or if there are specific approaches for this scenario. Real-Time Processing: I'm also concerned about the real-time aspect of the analysis. How can I ensure that the sentiment analysis is performed quickly enough to provide insights in real-time without introducing significant delays? Question: I'm seeking guidance on the best approach to implement real-time sentiment analysis on live audio streams using Python. Are there any specialized libraries or techniques for audio-based sentiment analysis that I should be aware of? How can I effectively process the audio data and perform sentiment analysis in real-time? Any insights, code examples, or recommended resources would be greatly appreciated.

Answer: The solution I found here by following these steps:  Adding middleware to make text from video Running the sentiment analysis for each complete sentence (Using Whisper as @doneforaiur suggested)  So, the latency here depends on the length of the words of the sentence. It's a trade-off for getting the analysis for the full context. Making chunks doesn't reflect the whole sentence thus the complete context. There is a possible way to extend this by adding multiple sentence feeds to the analysis system to get deeper context. I'm thinking about it. Sometimes, we describe a scenario with multiple sentences. Thanks, @doneforaiur for setting me in the right direction.
Question: I want to read a xml file generated by komoot into a DataFrame. Here is the structure of the xml file: <?xml version='1.0' encoding='UTF-8'?> <gpx version="1.1" creator="https://www.komoot.de" xmlns="http://www.topografix.com/GPX/1/1" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://www.topografix.com/GPX/1/1 http://www.topografix.com/GPX/1/1/gpx.xsd"> <metadata> <name>Title</name> <author> <link href="https://www.komoot.de"> <text>komoot</text> <type>text/html</type> </link> </author> </metadata> <trk> <name>Title</name> <trkseg> <trkpt lat="60.126749" lon="4.250254"> <ele>455.735013</ele> <time>2023-08-20T17:42:34.674Z</time> </trkpt> <trkpt lat="60.126580" lon="4.250247"> <ele>455.735013</ele> <time>2023-08-20T17:42:36.695Z</time> </trkpt> <trkpt lat="60.126484" lon="4.250240"> <ele>455.735013</ele> <time>2023-08-20T17:44:15.112Z</time> </trkpt> </trkseg> </trk> </gpx>  I tried this code: pd.read_xml('testfile.gpx',xpath='./gpx/trk/trkseg')  But somehow it seems there are problems with my xpath. Namely, I get this ValueError: ValueError: xpath does not return any nodes. Be sure row level nodes are in xpath. If document uses namespaces denoted with xmlns, be sure to define namespaces and use them in xpath.  I tried a lot but no xpath I chose worked out.

Answer: Following the ValueError guidelines, you need to pass a namespace to read_xml : df = ( pd.read_xml( "testfile.gpx", xpath=".//doc:trkseg/doc:trkpt", namespaces={"doc": "http://www.topografix.com/GPX/1/1"} ) )  Output : print(df) lat lon ele time 0 60.126749 4.250254 455.735013 2023-08-20T17:42:34.674Z 1 60.126580 4.250247 455.735013 2023-08-20T17:42:36.695Z 2 60.126484 4.250240 455.735013 2023-08-20T17:44:15.112Z
Question: l = [1,2,3,4] popping the last element would be an O(1) operation since:  It returns the last element Changes a few fixed attributes like len of the list  Why can't we do the same with pop(0)?  Return the first element Change the pointer of the first element (index 0) to that of the second element (index 1) and change a few fixed attributes? 

Answer: Lists could have been implemented to do what you suggest, but it would add complexity and overhead, for an operation better handled by collections.deque. All lists everywhere would need extra metadata to track how much empty space is at the front (important for handling the resize policy, and calling free on the right pointer when the list dies or gets resized), and the logic for when and how to resize would also become more complicated. The tradeoff was not deemed worthwhile.
Question: I'm trying to run a chain in LangChain with memory and multiple inputs. The closest error I could find was was posted here, but in that one, they are passing only one input. Here is the setup: from langchain.llms import OpenAI from langchain.chains import LLMChain from langchain.prompts import PromptTemplate from langchain.memory import ConversationBufferMemory llm = OpenAI( model="text-davinci-003", openai_api_key=environment_values["OPEN_AI_KEY"], # Used dotenv to store API key temperature=0.9, client="", ) memory = ConversationBufferMemory(memory_key="chat_history") prompt = PromptTemplate( input_variables=[ "text_one", "text_two", "chat_history" ], template=( """You are an AI talking to a huamn. Here is the chat history so far: {chat_history} Here is some more text: {text_one} and here is a even more text: {text_two} """ ) ) chain = LLMChain( llm=llm, prompt=prompt, memory=memory, verbose=False )  When I run output = chain.predict( text_one="Hello", text_two="World" )  I get ValueError: One input key expected got ['text_one', 'text_two'] I've looked at this stackoverflow post, which suggests to try: output = chain( inputs={ "text_one" : "Hello", "text_two" : "World" } )  which gives the exact same error. In the spirit of trying different things, I've also tried: output = chain.predict( # Also tried .run() here inputs={ "text_one" : "Hello", "text_two" : "World" } )  which gives Missing some input keys: {'text_one', 'text_two'}. I've also looked at this issue on the langchain GitHub, which suggests to do pass the llm into memory, i.e. # Everything the same except... memory = ConversationBufferMemory(llm=llm, memory_key="chat_history") # Note the llm here  and I still get the same error. If someone knows a way around this error, please let me know. Thank-you.

Answer: While drafting this question, I came across the answer. When defining the memory variable, pass an input_key="human_input" and make sure each prompt has a human_input defined. memory=ConversationBufferMemory( memory_key="chat_history", input_key="human_input" )  Then, in each prompt, make sure there is a human_input input. prompt = PromptTemplate( input_variables=[ "text_one", "text_two", "chat_history", "human_input", # Even if it's blank ], template=( """You are an AI talking to a huamn. Here is the chat history so far: {chat_history} Here is some more text: {text_one} and here is a even more text: {text_two} {human_input} """ ) )  Then, build your chain: chain = LLMChain( llm=llm, prompt=prompt, memory=memory, # Contains the input_key verbose=False )  And then run it as: output = chain.predict( human_input="", # or whatever you want text_one="Hello", text_two="World" ) print(output) # On my machine, it outputs: '\nAI: Hi there! How can I help you?'
Question: PEP 622 introduced match statement as an alternative to if-elif-else. However, one thing I can't find in the proposal or in any of the material online is whether the match statement can be used as an expression and not just as a statement. A couple of examples to make it clear: Example 1: def make_point_2d(pt): match pt: case (x, y): return Point2d(x, y) case _: raise TypeError("not a point we support")  Example 2: match response.status: case 200: do_something(response.data) case 301 | 302: retry(response.location)  In the first example, the function returns from inside a case clause, and in the second example, nothing is returned. But I want to be able to do something like the following hypothetical example: spouse = match name: case "John": "Jane" case "David": "Alice" print(spouse)  But it doesn't compile.

Answer: Not in Python. In Rust and Haskell, matches are expressions composed of expressions: let spouse = match name { // expr => expr, "John" => "Jane", // expr => {stmt; stmt; expr}, "David" => {let s = "Alice"; println!("Matched David"); s}, _ => panic!("Unknown name"), };  do spouse <- case name of "John" -> return "Jane" "David" -> do let s = "Alice" putStrLn "Matched David" return s  ...so it's not technically impossible that the Python match statement couldn't have been designed to work as an expression. Presumably, the reason it was avoided for Python are syntactical reasons:  Python does not have syntax for expression statements (e.g. {stmt; stmt; expr}). spouse = match name: case "John": "Jane" case "David": # stmt; stmt; expr s = "Alice" print("Matched David") s   Indentation to indicate a block being treated as an expression may look unnatural to some people. (See also: match grammar.)   That said, it is possible your proposed syntax could be accepted in the future.