eshangj/stackoverflow_q_and_a_sample huggingface dataset:
Question: I have a use case that requires passing a dictionary in a try/exception clause in Python 3.x The error message can be accessed as a string using str() function, but I can't figure out who to get it as a dictionary. try: raise RuntimeError({'a':2}) except Exception as e: error = e print(error['a'])  e is a RuntimeError object and I can't find any method that returns the message in its original format.

Answer: Exceptions store their init args in an "args" attribute: try: raise RuntimeError({'a':2}) except Exception as e: (the_dict,) = e.args print(the_dict["a"])  That being said, if you want an exception type which has a structured key/value context associated, it would be best to define your own custom exception subclass for this purpose rather than re-use the standard library's RuntimeError directly. That's because if you catch such an exception and attempt to unpack the dictionary context, you would need to detect and handle your RuntimeError instances differently from RuntimeError instances that the standard library may have raised. Using a different type entirely will make it much cleaner and easier to distinguish these two cases in your code.
Question: I use a ThreadPoolExecutor to quickly check a list of proxies to see which ones are dead or alive. with concurrent.futures.ThreadPoolExecutor(max_workers=5) as executor: futures = [] for proxy in proxies: future = executor.submit(is_proxy_alive, proxy) futures.append(future) for future in futures: print(future.result()) # prints true or false depending on if proxy is alive. # how do I get the specific proxy I passed in the arguments # so that I can make a dictionary here?  My goal is to get the argument (the proxy) I passed to the executor when iterating through the results to know which exact proxies are dead or alive, so I could make a dictionary that might look like this: {"IP1": False, "IP2": True, "IP3": True} One method I can think of is returning the proxy I sent on top of returning true/false, but is there a better way to do it externally so the function doesn't have to return more than just a bool?

Answer: While submitting the task, you could create a mapping from future to its proxy. with concurrent.futures.ThreadPoolExecutor(max_workers=5) as executor: future_proxy_mapping = {} futures = [] for proxy in proxies: future = executor.submit(is_proxy_alive, proxy) future_proxy_mapping[future] = proxy futures.append(future) for future in futures: proxy = future_proxy_mapping[future] print(proxy) print(future.result())
Question: I'm trying to speed up some code that I have written in Python and have thought about writing some code in C to do so. However, I am using PyCharm and its supported languages section, https://www.jetbrains.com/help/pycharm/supported-languages.html doesn't mention C, meaning no option to just create a C file and then import it into python. Is there any other way of doing it?

Answer: Unfortunately PyCharm does not support any C/C++ coding and there are no existing plugins for PyCharm that will support this. With that said, there is an IDE for C and C++ called CLion which is released by JetBrains just like PyCharm. CLion supports many features varying from Python debugger to Python Console for working with python and also supports all features that Pycharm offers. This should satisfy your needs towards using C code with Python in the most healthy way. Here are some useful links for you: https://www.jetbrains.com/clion/ https://www.jetbrains.com/help/clion/python.html#features Good luck!
Question: I'm trying to plat marker where maker opacity is changed by some vector. But marker edge color opacity is constant.  fig.add_trace(go.Scatter(x=real.index, y=real['some_value'], mode='markers', marker={'opacity': real['another value'], 'color':'green', 'size':10, 'line':dict(width=1, color='rgba(165,42,42,1)')} ))  It can be seen in plot below, that the marker edge color opacity, is changed together with color opacity of the marker filling. My purpose is to keep line (marker edge ) opacity constant.  NOTE: this question doesn't answer the question: plotly.py: change line opacity, leave markers opaque

Answer: You can easily rescale a pandas series between 0 and 1 and use that as an argument in rgba(red,green,blue,opacity) like color='rgba(100,0,255,'+opac+')' where opac is some opacity between 0 and 1 for a certain marker in your figure. The color property of the markers is unique for any go.Scatter(), so you'll have to add a unique trace for each point. Then, at the same time, you can set color (and opacity if you should feel so inclined) for the outline of the marker using marker=dict(line=dict(color='rgba(100,0,255,1)')) In the figure below, I've set the outline color as 'rgba(100,0,255,1)', and the opacity of the marker fill as varying according to the logic above. This way, the highest values will appear as a completely "filled" marker:  But you can also set a more clearly defined line using, for example, line=dict(color='rgba(0,0,0,1)', width = 2) to get something like this:  Now you can play around with all the rgba arguments to find a color to your liking. Complete code: # imports import plotly.graph_objects as go import pandas as pd import numpy as np # sample data in the form of an hourlt np.random.seed(1234) tseries = pd.date_range("01.01.2020", "01.04.2020", freq="H") data = np.random.randint(-100, 100, size=(len(tseries), 3)) df = pd.DataFrame(data=data) df.columns=list('ABC') df['C_scaled'] = df['C'].max()/df['C'] df['C_scaled'] = (df['C']-df['C'].min())/(df['C'].max()-df['C'].min()) df = df.sort_values(by=['C_scaled'], ascending=False) fig=go.Figure() for ix in df.index: d = df.iloc[ix] opac = str(d['C_scaled']) fig.add_trace(go.Scatter(x=[d['A']], y=[d['B']], showlegend=False, marker=dict(size = 14, color='rgba(100,0,255,'+opac+')', line=dict(color='rgba(0,0,0,1)', width = 2))) ) fig.show()  Edit: Hoverinfo side-effects Just include the following to edit the hoverinfo so that x and y values always are shown on hover to the closeset values: fig.update_layout(hovermode="x") fig.update_traces(hoverinfo = 'x+y')
Question: I'm using the librosa library to get and filter spectrograms from audio data. I mostly understand the math behind generating a spectrogram:  Get signal window signal for each window compute Fourier transform Create matrix whose columns are the transforms Plot heat map of this matrix  So that's really easy with librosa: spec = np.abs(librosa.stft(signal, n_fft=len(window), window=window)  Yay! I've got my matrix of FFTs. Now I see this function librosa.amplitude_to_db and I think this is where my ignorance of signal processing starts to show. Here is a snippet I found on Medium: spec = np.abs(librosa.stft(y, hop_length=512)) spec = librosa.amplitude_to_db(spec, ref=np.max)  Why does the author use this amplitude_to_db function? Why not just plot the output of the STFT directly?

Answer: The range of perceivable sound pressure is very wide, from around 20 μPa (micro Pascal) to 20 Pa, a ratio of 1 million. Furthermore the human perception of sound levels is not linear, but better approximated by a logarithm. By converting to decibels (dB) the scale becomes logarithmic. This limits the numerical range, to something like 0-120 dB instead. The intensity of colors when this is plotted corresponds more closely to what we hear than if one used a linear scale. Note that the reference (0 dB) point in decibels can be chosen freely. The default for librosa.amplitude_to_db is to compute numpy.max, meaning that the max value of the input will be mapped to 0 dB. All other values will then be negative. The function also applies a threshold on the range of sounds, by default 80 dB. So anything lower than -80 dB will be clipped -80 dB.
Question: I have pivoted the Customer ID against their most frequently purchased genres of performances: Genre Jazz Dance Music Theatre Customer 100000000001 0 3 1 2 100000000002 0 1 6 2 100000000003 0 3 13 4 100000000004 0 5 4 1 100000000005 1 10 16 14  My desired result is to append the column names according to the rankings: Genre Jazz Dance Music Theatre Rank1 Rank2 Rank3 Rank4 Customer 100000000001 0 3 1 2 Dance Theatre Music Jazz 100000000002 0 1 6 2 Music Theatre Dance Jazz 100000000003 0 3 13 4 Music Theatre Dance Jazz 100000000004 0 5 4 1 Dance Music Theatre Jazz 100000000005 1 10 16 14 Music Theatre Dance Jazz  I have looked up some threads but the closest thing I can find is idxmax. However that only gives me Rank1. Could anyone help me to get the result I need? Thanks a lot! Dennis

Answer: Use: i = np.argsort(df.to_numpy() * -1, axis=1) r = pd.DataFrame(df.columns[i], index=df.index, columns=range(1, i.shape[1] + 1)) df = df.join(r.add_prefix('Rank'))  Details: Use np.argsort along axis=1 to get the indices i that would sort the genres in descending order. print(i) array([[1, 3, 2, 0], [2, 3, 1, 0], [2, 3, 1, 0], [1, 2, 3, 0], [2, 3, 1, 0]])  Create a new dataframe r from the columns of dataframe df taken along the indices i (i.e df.columns[i]), then use DataFrame.join to join the dataframe r with df: print(df) Jazz Dance Music Theatre Rank1 Rank2 Rank3 Rank4 Customer 100000000001 0 3 1 2 Dance Theatre Music Jazz 100000000002 0 1 6 2 Music Theatre Dance Jazz 100000000003 0 3 13 4 Music Theatre Dance Jazz 100000000004 0 5 4 1 Dance Music Theatre Jazz 100000000005 1 10 16 14 Music Theatre Dance Jazz
Question: I am writing a CSV file with the following function: import csv import os import aiofiles async def write_extract_file(output_filename: str, csv_list: list): """ Write the extracted content into the file """ try: async with aiofiles.open(output_filename, "w+") as csv_file: writer = csv.DictWriter(csv_file, fieldnames=columns.keys()) writer.writeheader() writer.writerows(csv_list) except FileNotFoundError: print("Output file not present", output_filename) print("Current dir: ", os.getcwd()) raise FileNotFoundError  However, as there is no await allowed over writerows method, there are no rows being written into the CSV file. How to resolve this issue? Is there any workaround available? Thank you. Entire code can be found here.

Answer: In my opinion it’s better not to try to use the aiofiles with the csv module and run the synchronous code using loop.run_in_executor and wait it asynchronously like below: def write_extract_file(output_filename: str, csv_list: list): """ Write the extracted content into the file """ try: with open(output_filename, "w+") as csv_file: writer = csv.DictWriter(csv_file, fieldnames=columns.keys()) writer.writeheader() writer.writerows(csv_list) except FileNotFoundError: print("Output file not present", output_filename) print("Current dir: ", os.getcwd()) raise FileNotFoundError async def main(): loop = asyncio.get_running_loop() await loop.run_in_executor(None, write_extract_file, 'test.csv', csv_list)
Question: I wonder how to check if a pandas dataframe has negative value in 1 or more columns and return only boolean value (True or False). Can you please help? In[1]: df = pd.DataFrame(np.random.randn(10, 3)) In[2]: df Out[2]: 0 1 2 0 -1.783811 0.736010 0.865427 1 -1.243160 0.255592 1.670268 2 0.820835 0.246249 0.288464 3 -0.923907 -0.199402 0.090250 4 -1.575614 -1.141441 0.689282 5 -1.051722 0.513397 1.471071 6 2.549089 0.977407 0.686614 7 -1.417064 0.181957 0.351824 8 0.643760 0.867286 1.166715 9 -0.316672 -0.647559 1.331545  Expected output:- Out[3]: True 

Answer: Actually, if speed is important, I did a few tests: df = pd.DataFrame(np.random.randn(10000, 30000))  Test 1, slowest: pure pandas (df < 0).any().any() # 303 ms ± 1.28 ms per loop (mean ± std. dev. of 7 runs, 1 loop each)  Test 2, faster: switching over to numpy with .values for testing the presence of a True entry (df < 0).values.any() # 269 ms ± 8.19 ms per loop (mean ± std. dev. of 7 runs, 1 loop each)  Test 3, maybe even faster, though not significant: switching over to numpy for the whole thing (df.values < 0).any() # 267 ms ± 1.48 ms per loop (mean ± std. dev. of 7 runs, 1 loop each)
Question: I'm trying to restore a pickled config file from RLLib (json didn't work as shown in this post), and getting the following error: config = pickle.load(open(f"{path}/params.pkl", "rb")) --------------------------------------------------------------------------- ValueError Traceback (most recent call last) <ipython-input-28-c964561b863c> in <module> ----> 1 config = pickle.load(open(f"{path}/params.pkl", "rb")) ValueError: unsupported pickle protocol: 5  Python Version = 3.7.0 How can I open this file in 3.7?

Answer: Use pickle5 or load it into python 3.8+ and then serialize it to a lower version of it using the protocol parameter.
Question: I recently had a SonarQube rule (https://rules.sonarsource.com/java/RSPEC-4784) bring to my attention some performance issues which could be used as a denial of service against a Java regular expression implementation. Indeed, the following Java test shows how slow the wrong regular expression can be:  import org.junit.Test; public class RegexTest { @Test public void fastRegex1() { "aaaaaaaaaaaaaaaaaaaaaaaaaaaabs".matches("(a+)b"); } @Test public void fastRegex2() { "aaaaaaaaaaaaaaaaaaaaaaaaaaaab".matches("(a+)+b"); } @Test public void slowRegex() { "aaaaaaaaaaaaaaaaaaaaaaaaaaaabs".matches("(a+)+b"); } }  As you can see, the first two tests are fast, the third one is incredibly slow (in Java 8)  The same data and regex in Perl or Python, however, is not at all slow, which leads me to wonder why it is that this regular expression is so slow to evaluate in Java. $ time perl -e '"aaaaaaaaaaaaaaaaaaaaaaaaaaaabs" =~ /(a+)+b/ && print "$1\n"' aaaaaaaaaaaaaaaaaaaaaaaaaaaa real 0m0.004s user 0m0.000s sys 0m0.004s $ time python3 -c 'import re; m=re.search("(a+)+b","aaaaaaaaaaaaaaaaaaaaaaaaaaaabs"); print(m.group(0))' aaaaaaaaaaaaaaaaaaaaaaaaaaaab real 0m0.018s user 0m0.015s sys 0m0.004s  What is it about the extra matching modifier + or trailing character s in the data which makes this regular expression so slow, and why is it only specific to Java?

Answer: Caveat: I don't really know much about regex internals, and this is really conjecture. And I can't answer why Java suffers from this, but not the others (also, it is substantially faster than your 12 seconds in jshell 11 when I run it, so it perhaps only affects certain versions). "aaaaaaaaaaaaaaaaaaaaaaaaaaaabs".matches("(a+)+b")  There are lots of ways that lots of as could match: (a)(a)(a)(a) (aa)(a)(a) (a)(aa)(a) (aa)(aa) (a)(aaa) etc.  For the input string "aaaaaaaaaaaaaaaaaaaaaaaaaaaab", it will greedily match all of those as in a single pass, match the b, job done. For "aaaaaaaaaaaaaaaaaaaaaaaaaaaabs", when it gets to the end and finds that the string doesn't match (because of the s), it's not correctly recognizing that the s means it can never match. So, having gone through and likely matched as (aaaaaaaaaaaaaaaaaaaaaaaaaaaa)bs  it thinks "Oh, maybe it failed because of the way I grouped the as - and goes back and tries all the other combinations of the as. (aaaaaaaaaaaaaaaaaaaaaaaaaaa)(a)bs // Nope, still no match (aaaaaaaaaaaaaaaaaaaaaaaaaa)(aa)bs // ... (aaaaaaaaaaaaaaaaaaaaaaaaa)(aaa)bs // ... ... (a)(aaaaaaaaaaaaaaaaaaaaaaaaaaa)bs // ... (aaaaaaaaaaaaaaaaaaaaaaaaaa(a)(a)bs // ... (aaaaaaaaaaaaaaaaaaaaaaaaa(aa)(a)bs // ... (aaaaaaaaaaaaaaaaaaaaaaaa(aaa)(a)bs // ... ...  There are lots of these (I think there are something like 2^27 - that's 134,217,728 - combinations for 28 as, because each a can either be part of the previous group, or start its own group), so it takes a long time.
Question: I'm trying to copy the value from the Excel file but it returns this error message: Exception while executing function: Functions.extract Result: Failure Exception: OSError: [Errno 30] Read-only file system: './data_download/xxxxx.xlsx' Stack: File "/azure-functions-host/workers/python/3.8/LINUX/X64/azure_functions_worker/dispatcher.py", line 343, in handle_invocation_request call_result = await self._loop.run_in_executor( File "/usr/local/lib/python3.8/concurrent/futures/thread.py", line 57, in run result = self.fn(*self.args, **self.kwargs) File "/azure-functions-host/workers/python/3.8/LINUX/X64/azure_functions_worker/dispatcher.py", line 480, in __run_sync_func return func(**params) File "/home/site/wwwroot/extract/_init_.py", line 62, in main with open(download_file_path, "wb") as download_file:  My code with open(download_file_path, "wb") as download_file: 

Answer: This is not related to Azure function, in general Only /tmp seems to be writable Try adding tmp to the file path filepath = '/tmp/' + key
Question: I would like to have a multiple line plot within same canvas tied with the same x-axis as shown something in the figure:  Using subplots does not achieve the intended desire. import plotly.express as px from plotly.subplots import make_subplots import plotly.graph_objects as go fig = make_subplots(rows=2, shared_xaxes=True,vertical_spacing=0.1) fig.add_scatter(y=[2, 1, 3], row=1, col=1) fig.add_scatter(y=[1, 3, 2], row=2, col=1) fig.show()  May I know how this can be done, appreciate if someone can point to good reading material

Answer: With a dataset such as this you can select any number of columns, set up a figure using fig = make_subplots() with shared_xaxes set to True and then add your series with a shared x-axis using fig.add_trace(go.Scatter(x=df[col].index, y=df[col].values), row=i, col=1) in a loop to get this:  Let me know if this is a setup you can use but need a little tweaking. Complete code: import plotly.graph_objects as go import plotly.io as pio from plotly.subplots import make_subplots import pandas as pd # data pio.templates.default = "plotly_white" df = pd.read_csv('https://raw.githubusercontent.com/plotly/datasets/master/finance-charts-apple.csv') df = df.set_index('Date') df.tail() cols = df.columns[:-4] ncols = len(cols) # subplot setup fig = make_subplots(rows=ncols, cols=1, shared_xaxes=True) for i, col in enumerate(cols, start=1): fig.add_trace(go.Scatter(x=df[col].index, y=df[col].values), row=i, col=1) fig.show()
Question: For example in Pandas I would do data_df = ( pd.DataFrame(dict(col1=['a', 'b', 'c'], col2=['1', '2', '3'])) .pipe(lambda df: df[df.col1 != 'a']) )  This is similar to R's pipe %>% Is there something similar in PySpark?

Answer: I think, in pyspark, you can easily achieve this pipe functionality with help of pipeline.  convert each of the pipe function into the transformer. There are some predefined transformers that spark provides, we can make use of that also Create pipeline using the transformers Run the pipeline to transform provided dataframe  Example: Let's take the example you provided Input Dataframe to transform  val df = Seq(("a", 1), ("b", 2), ("c", 3)).toDF("col1", "col2") df.show(false) df.printSchema() /** * +----+----+ * |col1|col2| * +----+----+ * |a |1 | * |b |2 | * |c |3 | * +----+----+ * * root * |-- col1: string (nullable = true) * |-- col2: integer (nullable = false) */  1. Convert each of the pipe function into the transformer for .pipe(lambda df: df[df.col1 != 'a']), we can easily use spark SQLTransformer. so no need to create custom transformer 2. Create pipeline using the transformers  val transform1 = new SQLTransformer() .setStatement("select * from __THIS__ where col1 != 'a'") val transform2 = new SQLTransformer() .setStatement("select col1, col2, SQRT(col2) as col3 from __THIS__") val pipeline = new Pipeline() .setStages(Array(transform1, transform2))  3. Run the pipeline to transform provided dataframe pipeline.fit(df).transform(df) .show(false) /** * +----+----+------------------+ * |col1|col2|col3 | * +----+----+------------------+ * |b |2 |1.4142135623730951| * |c |3 |1.7320508075688772| * +----+----+------------------+ */
Question: I need to extract item combination from 2 lists by means of python Spacy Matcher. The problem is following: Let us have 2 lists: colors=['red','bright red','black','brown','dark brown'] animals=['fox','bear','hare','squirrel','wolf']  I match the sequences by the following code: first_color=[] last_color=[] only_first_color=[] for color in colors: if ' ' in color: first_color.append(color.split(' ')[0]) last_color.append(color.split(' ')[1]) else: only_first_color.append(color) matcher = Matcher(nlp.vocab) pattern1 = [{"TEXT": {"IN": only_first_color}},{"TEXT":{"IN": animals}}] pattern2 = [{"TEXT": {"IN": first_color}},{"TEXT": {"IN": last_color}},{"TEXT":{"IN": animals}}] matcher.add("ANIMALS", None, pattern1,pattern2) doc = nlp('bright red fox met black wolf') matches = matcher(doc) for match_id, start, end in matches: string_id = nlp.vocab.strings[match_id] # Get string representation span = doc[start:end] # The matched span print(start, end, span.text)  It gives the output: 0 3 bright red fox 1 3 red fox 4 6 black wolf  How can i extract only 'bright red fox' and 'black wolf'? Should i change the patterns rules or post-process the matches? Any thoughts appreciate!

Answer: You may use spacy.util.filter_spans:  Filter a sequence of Span objects and remove duplicates or overlaps. Useful for creating named entities (where one token can only be part of one entity) or when merging spans with Retokenizer.merge. When spans overlap, the (first) longest span is preferred over shorter spans.  Python code: matches = matcher(doc) spans = [doc[start:end] for _, start, end in matches] for span in spacy.util.filter_spans(spans): print(span.start, span.end, span.text)  Output: 0 3 bright red fox 4 6 black wolf
Question: I'm a beginner to Pytorch and wanted to type this statement as a whole if else statement:- torch.device('cuda' if torch.cuda.is_available() else 'cpu')  Can somebody help me?

Answer: Here is the code as a whole if-else statement: torch.device('cuda' if torch.cuda.is_available() else 'cpu')  if torch.cuda.is_available(): torch.device('cuda') else: torch.device('cpu')  Since you probably want to store the device for later, you might want something like this instead: device = torch.device('cuda' if torch.cuda.is_available() else 'cpu')  if torch.cuda.is_available(): device = torch.device('cuda') else: device = torch.device('cpu')   Here is a post and discussion about the ternary operator in Python: https://stackoverflow.com/a/2802748/13985765 From that post: value_when_true if condition else value_when_false
Question: I have a problem here. I don't know why this code does not work. newline = '\n' tasks_choosen = ['markup', 'media', 'python_api', 'script', 'style', 'vue'] print(f'{ newline }### Initializing project with the following tasks: { ' '.join(tasks_choosen) }.{ newline }')  Error:  File "new-gulp-project.py", line 85 print(f'{ newline }### Initializing project with the following tasks: { ' '.join(tasks_choosen) }.{ newline }') SyntaxError: f-string: expecting '}'  Can anyone help me? Thanks

Answer: Because you use single quotes twice you get: print(f'{ newline }### Initializing project with the following tasks: { ' instead of print(f'{ newline }### Initializing project with the following tasks: { ' '.join(tasks_choosen) }.{ newline }')  Use double quotes inside: print(f'{ newline }### Initializing project with the following tasks: { " ".join(tasks_choosen) }.{ newline }')