eshangj/stackoverflow_q_and_a_sample huggingface dataset:
Question: I want to use a user-provided string as a filename for exporting, but have to make sure that the string is permissible on my system as a filename. From my side it would be OK to replace any forbidden character with e.g. '_'.  Here I found a list of forbidden characters for filenames. It should be easy enough to use the str.replace() function, I was just wondering if there is already something out there that does that, potentially even taking into account what OS I am on.

Answer:  pathvalidate is a Python library to sanitize/validate a string such as filenames/file-paths/etc.  This library provides both utilities for validation of paths: import sys from pathvalidate import ValidationError, validate_filename try: validate_filename("fi:l*e/p\"a?t>h|.t<xt") except ValidationError as e: print("{}\n".format(e), file=sys.stderr)  And utilities for sanitizing paths: from pathvalidate import sanitize_filename fname = "fi:l*e/p\"a?t>h|.t<xt" print("{} -> {}".format(fname, sanitize_filename(fname)))
Question: Problem I was following a Tensorflow 2 tutorial on how to load images with pure Tensorflow, because it is supposed to be faster than with Keras. The tutorial ends before showing how to split the resulting dataset (~tf.Dataset) into a train and validation dataset.  I checked the reference for tf.Dataset and it does not contain a split() method. I tried slicing it manually but tf.Dataset neither contains a size() nor a length() method, so I don't see how I could slice it myself. I can't use the validation_split argument of Model.fit() because I need to augment the training dataset but not the validation dataset.  Question What is the intended way to split a tf.Dataset or should I use a different workflow where I won't have to do this? Example Code (from the tutorial) BATCH_SIZE = 32 IMG_HEIGHT = 224 IMG_WIDTH = 224 list_ds = tf.data.Dataset.list_files(str(data_dir/'*/*')) def get_label(file_path): # convert the path to a list of path components parts = tf.strings.split(file_path, os.path.sep) # The second to last is the class-directory return parts[-2] == CLASS_NAMES def decode_img(img): # convert the compressed string to a 3D uint8 tensor img = tf.image.decode_jpeg(img, channels=3) # Use `convert_image_dtype` to convert to floats in the [0,1] range. img = tf.image.convert_image_dtype(img, tf.float32) # resize the image to the desired size. return tf.image.resize(img, [IMG_WIDTH, IMG_HEIGHT]) def process_path(file_path): label = get_label(file_path) # load the raw data from the file as a string img = tf.io.read_file(file_path) img = decode_img(img) return img, label labeled_ds = list_ds.map(process_path, num_parallel_calls=AUTOTUNE) #... #...  I can either split list_ds (list of files) or labeled_ds (list of images and labels), but how?

Answer: I don't think there's a canonical way (typically, data is being split e.g. in separate directories). But here's a recipe that will let you do it dynamically: # Caveat: cache list_ds, otherwise it will perform the directory listing twice. ds = list_ds.cache() # Add some indices. ds = ds.enumerate() # Do a rougly 70-30 split. train_list_ds = ds.filter(lambda i, data: i % 10 < 7) test_list_ds = ds.filter(lambda i, data: i % 10 >= 7) # Drop indices. train_list_ds = train_list_ds.map(lambda i, data: data) test_list_ds = test_list_ds.map(lambda i, data: data)
Question: I'm parsing CLI arguments in my program with the argparse library. I would like to parse an argument that can repeat, with the following behaviour:  if the argument appears at least once, its values are stored in a list, if the argument doesn't appear, the value is some default list.  I have the following code so far: import argparse ap = argparse.ArgumentParser(description="Change channel colours.") ap.add_argument('-c', '--channel', action='append', default=['avx', 'fbx']) print(ap.parse_known_args(['-c', 'iasdf', '-c', 'fdas'])) print(ap.parse_known_args())  This appropriately sets a default list, however it doesn't start with an empty list when the argument appears. In other words, the second print statement prints the correct value (the default list), but the first one prints ['avx', 'fbx', 'iasdf', 'fdas']  instead of ['iasdf', 'fdas']  Is there a way in argparse to do what I want without doing something like if len(args.channel) > 2: args.channel = args.channel[2:]  after the fact?

Answer: There's a bug/issue discussing this behavior. I wrote several posts to that. https://bugs.python.org/issue16399 argparse: append action with default list adds to list instead of overriding For now the only change is in documentation, not in behavior. All defaults are placed in the namespace at the start of parsing. For ordinary actions, user values overwrite the default. But in the append case, they are just added to what's there already. It doesn't try to distinguish between values placed by the default, and previous user values. I think the simplest solution is to leave the default as is, and check after parsing for None or empty list (I don't recall which), and insert your default. You don't get extra points for doing all the parsing in argparse. A bit of post parsing processing is quite ok.
Question: I'm using Holoviews to construct a dashboard of charts. Some of these charts have percentages in the y axis where as others have sums/counts etc. When I try to output all the charts I have created to a html file, all the charts change their y axis to match the axis of the first chart of my chart list.  For example:  Chart 1 is a sum, values go from 0 to 1000 Chart 2 is a % Chart 3 is a %  when I combine these charts in holoviews using:  Charts = Chart 1 + Chart 2 + Chart 3  The y axis of charts 2 and 3 become the same as chart 1. Does anyone know why this is happening and how I can fix it so all the charts keep their individual axis pertinent to what they are trying to represent.  Thank you!

Answer: This happens when the y-axes have the same name. You need to use option axiswise=True if you want every plot to get its own independent x-axis and y-axis.  There's a short reference to axiswise in the holoviews FAQ: https://www.holoviews.org/FAQ.html  Here's a code example that I've checked and works: # import libraries etc. import numpy as np import pandas as pd import holoviews as hv from holoviews import opts hv.extension('bokeh') # create some sample data df1 = pd.DataFrame({ 'x': np.random.rand(10), 'y': np.random.rand(10), }) df2 = pd.DataFrame({ 'x': np.random.rand(10) * 10, 'y': np.random.rand(10) * 10, }) # set axiswise=True so that every plot gets its own independent x- and y-axis plot1 = hv.Scatter(df1).opts(axiswise=True) plot2 = hv.Scatter(df2).opts(axiswise=True) plot1 + plot2  Or alternatively you can do: plot1 = hv.Scatter(df1) plot2 = hv.Scatter(df2) (plot1 + plot2).opts(opts.Scatter(axiswise=True))   If this doesn't work when you try my code example, you may have to upgrade to the latest version of holoviews. This can be done as follows: Install the latest git versions of holoviews, hvplot, panel, datashader and param
Question: The usage of {} in Python f-strings is well known to execute pieces of code and give the result in string format. However, what does the '=' at the end of the expression mean? log_file = open("log_aug_19.txt", "w") console_error = '...stuff...' # the real code generates it with regex log_file.write(f'{console_error=}') 

Answer: This is actually a brand-new feature as of Python 3.8.  Added an = specifier to f-strings. An f-string such as f'{expr=}' will expand to the text of the expression, an equal sign, then the representation of the evaluated expression.  Essentially, it facilitates the frequent use-case of print-debugging, so, whereas we would normally have to write: f"some_var={some_var}"  we can now write: f"{some_var=}"  So, as a demonstration, using a shiny-new Python 3.8.0 REPL: >>> print(f"{foo=}") foo=42 >>>
Question: Suppose I have an example of numpy array: import numpy as np X = np.array([2,5,0,4,3,1])  And I also have a list of arrays, like: A = [np.array([-2,0,2]), np.array([0,1,2,3,4,5]), np.array([2,5,4,6])]  I want to leave only these items of each list that are also in X. I expect also to do it in a most efficient/common way. Solution I have tried so far:  Sort X using X.sort(). Find locations of items of each array in X using: locations = [np.searchsorted(X, n) for n in A]  Leave only proper ones: masks = [X[locations[i]] == A[i] for i in range(len(A))] result = [A[i][masks[i]] for i in range(len(A))]   But it doesn't work because locations of third array is out of bounds: locations = [array([0, 0, 2], dtype=int64), array([0, 1, 2, 3, 4, 5], dtype=int64), array([2, 5, 4, 6], dtype=int64)]  How to solve this issue? Update I ended up with idx[idx==len(Xs)] = 0 solution. I've also noticed two different approaches posted between the answers: transforming X into set vs np.sort. Both of them has plusses and minuses: set operations uses iterations which is quite slow in compare with numpy methods; however np.searchsorted speed increases logarithmically unlike acceses of set items which is instant. That why I decided to compare performance using data with huge sizes, especially 1 million items for X, A[0], A[1], A[2].  

Answer: One idea would be less compute and minimal work when looping. So, here's one with those in mind - a = np.concatenate(A) m = np.isin(a,X) l = np.array(list(map(len,A))) a_m = a[m] cut_idx = np.r_[0,l.cumsum()] l_m = np.add.reduceat(m,cut_idx[:-1]) cl_m = np.r_[0,l_m.cumsum()] out = [a_m[i:j] for (i,j) in zip(cl_m[:-1],cl_m[1:])]  Alternative #1 : We can also use np.searchsorted to get the isin mask, like so - Xs = np.sort(X) idx = np.searchsorted(Xs,a) idx[idx==len(Xs)] = 0 m = Xs[idx]==a  Another way with np.intersect1d If you are looking for the most common/elegant one, think it would be with np.intersect1d - In [43]: [np.intersect1d(X,A_i) for A_i in A] Out[43]: [array([0, 2]), array([0, 1, 2, 3, 4, 5]), array([2, 4, 5])]   Solving your issue You can also solve your out-of-bounds issue, with a simple fix - for l in locations: l[l==len(X)]=0
Question: When having a Pandas DataFrame like this:  import pandas as pd import numpy as np df = pd.DataFrame({'today': [['a', 'b', 'c'], ['a', 'b'], ['b']], 'yesterday': [['a', 'b'], ['a'], ['a']]})   today yesterday 0 ['a', 'b', 'c'] ['a', 'b'] 1 ['a', 'b'] ['a'] 2 ['b'] ['a'] ... etc  But with about 100 000 entries, I am looking to find the additions and removals of those lists in the two columns on a row-wise basis. It is comparable to this question: Pandas: How to Compare Columns of Lists Row-wise in a DataFrame with Pandas (not for loop)? but I am looking at the differences, and Pandas.apply method seems not to be that fast for such many entries. This is the code that I am currently using. Pandas.apply with numpy's setdiff1d method: additions = df.apply(lambda row: np.setdiff1d(row.today, row.yesterday), axis=1) removals = df.apply(lambda row: np.setdiff1d(row.yesterday, row.today), axis=1)  This works fine, however it takes about a minute for 120 000 entries. So is there a faster way to accomplish this?

Answer: Not sure about performance, but at the lack of a better solution this might apply: temp = df[['today', 'yesterday']].applymap(set) removals = temp.diff(periods=1, axis=1).dropna(axis=1) additions = temp.diff(periods=-1, axis=1).dropna(axis=1)  Removals:  yesterday 0 {} 1 {} 2 {a}  Additions:  today 0 {c} 1 {b} 2 {b}
Question: I am using simpleSalesforce library for python to query SalesForce. I am looking at two different object in SalesForce: Account and Opportunity (parent-child). There is an accountId inside the opportunity object. I am trying to perform an inner join between the two and select the results (fields from both objects). a normal SQL statement would look like this: SELECT acc.Name, opp.StageName FROM Account AS acc JOIN Opportunity AS opp ON acc.Id = opp.AccountId  I am not sure how to translate this kind of query into SOQL.

Answer: Salesforce doesn't allow arbitrary joins. You must write relationship queries to traverse predefined relationships in the Salesforce schema. Here, you'd do something like SELECT Name, (SELECT StageName FROM Opportunities) FROM Account  No explicit join logic is required, or indeed permitted. Note too that your return values will be structured, nested JSON objects - Salesforce does not return flat rows like a SQL query would.
Question: The diagrams I am generating have many lines, and I want to automatically use colors and markers to distinguish them.  I tried this:  for i,studyDframeTuple in enumerate(studyDframeTuples): time = studyDframeTuple[1]['time'] error = studyDframeTuple[1]['Linf velocity error'] caseName = studyDirs[studyDframeTuple[0]] ax.plot(time, error, marker = i % 12, label=caseName)  Which circulates marker over (0,11). This kind of works, because for some reason marker < 12. When I use marker = i % 20, I get an error that makerstyle 12 is unknown.  This is an example of the diagram I'm generating, it's not pretty, it's only used for checking test results:    The diagrams are resulting from tests with varying parameters, hence the need to iterate over all available colors, line styles and markers, to make sure that when I have 100 lines on a diagram, I can distinguish the ones that belong to exploded solutions (values like 1e15 on this plot).  How can I put all markers in matplotib in a list and iterate over them?  Edit:  I hacked a list of my own like this mStyles = [".",",","o","v","^","<",">","1","2","3","4","8","s","p","P","*","h","H","+","x","X","D","d","|","_",0,1,2,3,4,5,6,7,8,9,10,11 ]  But what when this changes? Can I obtain this list programmatically from matplotlib?

Answer: 12 doesn't exist as marker value. You can have a dict of all existing markers using this : from matplotlib.lines import Line2D print(Line2D.markers)  Output: {'.': 'point', ',': 'pixel', 'o': 'circle', 'v': 'triangle_down', '^': 'triangle_up', '<': 'triangle_left', '>': 'triangle_right', '1': 'tri_down', '2': 'tri_up', '3': 'tri_left', '4': 'tri_right', '8': 'octagon', 's': 'square', 'p': 'pentagon', '*': 'star', 'h': 'hexagon1', 'H': 'hexagon2', '+': 'plus', 'x': 'x', 'D': 'diamond', 'd': 'thin_diamond', '|': 'vline', '_':'hline', 'P': 'plus_filled', 'X': 'x_filled', 0: 'tickleft', 1: 'tickright', 2: 'tickup', 3:'tickdown', 4: 'caretleft', 5: 'caretright', 6: 'caretup', 7: 'caretdown', 8: 'caretleftbase', 9: 'caretrightbase', 10: 'caretupbase', 11: 'caretdownbase', 'None': 'nothing', None: 'nothing', ' ': 'nothing', '': 'nothing'}
Question: I have just tried to update my anaconda environment to the latest version and I am now receiving errors. I opened the conda environment as an admin, and the commands issued were: conda update conda conda update anaconda  First command finished fine. Second command produced error: pythonw.exe - Entry Point Not Found The procedure entry point ?PyWinObject_FromULARGE_INTEGER@@YAPEAU_object@@AEAT_ULARGE_INTEGER@@@Z could not be located in the dynamic link library c:\ProgramData\Anaconda3\pythoncom37.dll   I have found a reference to this sort of error that requires me to copy a file libssl-1-1-x64.dll from Anaconda3/Library/bin with the one from Anaconda3/DLLs.  How to Fix Entry Point Not Found while installing libraries in conda environment However, I do not have that file, in the source location. Is there any commands I can issue to download this file again, or somewhere online I can safely download that one file from?

Answer: Sorry all - the clue was in the error message. The entry on how to fix entry point led me in the right direction. but it was the pythoncom37.dll file I needed to copy. That's what you get for blindly following instructions. Many thanks.
Question: Here's my current code values = pd.Series([False, False, True, True]) v_counts = values.value_counts() fig = plt.figure() plt.pie(v_counts, labels=v_counts.index, autopct='%.4f', shadow=True);  Currently, it shows only the percentage (using autopct) I'd like to present both the percentage and the actual value (I don't mind about the position)

Answer: Create your own formatting function. Note that you have to recalculate the actual value from the percentage in that function somehow def my_fmt(x): print(x) return '{:.4f}%\n({:.0f})'.format(x, total*x/100) values = pd.Series([False, False, True, True, True, True]) v_counts = values.value_counts() total = len(values) fig = plt.figure() plt.pie(v_counts, labels=v_counts.index, autopct=my_fmt, shadow=True);
Question: I have this dataframe: +------+--------------+------------+ | ID | Education | Score | +------+--------------+------------+ | 1 | High School | 7.884 | | 2 | Bachelors | 6.952 | | 3 | High School | 8.185 | | 4 | High School | 6.556 | | 5 | Bachelors | 6.347 | | 6 | Master | 6.794 | +------+--------------+------------+  I want to create a new column which is categorizing the score column. I want to label it as: 'bad', 'good', 'very good'. Which maybe would look like this:  +------+--------------+------------+------------+ | ID | Education | Score | Labels | +------+--------------+------------+------------+ | 1 | High School | 7.884 | Good | | 2 | Bachelors | 6.952 | Bad | | 3 | High School | 8.185 | Very good | | 4 | High School | 6.556 | Bad | | 5 | Bachelors | 6.347 | Bad | | 6 | Master | 6.794 | Bad | +------+--------------+------------+------------+  How can I do that? Thanks in advance

Answer: import pandas as pd # initialize list of lists data = [[1,'High School',7.884], [2,'Bachelors',6.952], [3,'High School',8.185], [4,'High School',6.556],[5,'Bachelors',6.347],[6,'Master',6.794]] # Create the pandas DataFrame df = pd.DataFrame(data, columns = ['ID', 'Education', 'Score']) df['Labels'] = ['Bad' if x<7.000 else 'Good' if 7.000<=x<8.000 else 'Very Good' for x in df['Score']] df ID Education Score Labels 0 1 High School 7.884 Good 1 2 Bachelors 6.952 Bad 2 3 High School 8.185 Very Good 3 4 High School 6.556 Bad 4 5 Bachelors 6.347 Bad 5 6 Master 6.794 Bad
Question: I am wondering, what the standard placement in a Python file for __all__? My assumption is directly below the import statements. However, I could not find this explicitly stated/asked anywhere. So, in general, where should one place __all__? Where would it be put in the below example file? #!/usr/bin/env python3 """Where to put __all__.""" from time import time # I think it should go here: __all__ = ["Hello"] SOME_GLOBAL = 0.0 class Hello: def __init__(self): pass if __name__ == "__main__": pass  Thank you in advance!

Answer: Per PEP 8:  Module level "dunders" (i.e. names with two leading and two trailing underscores) such as __all__, __author__, __version__, etc. should be placed after the module docstring but before any import statements except from __future__ imports.  So if you're following PEP 8 strictly, you were close. In practice, it's not obeyed strictly. A ton of the Python "included batteries" modules define __all__ after the imports, so your approach is perfectly fine.
Question: I am trying to use virtualenv to create a virtual python environment on my mac. I have downloaded virtualenv however I can't run it because it can't find the path to my installation of python3 even though I am supplying the correct path. Here is the command I have run and the response: virtualenv --python=/usr/local/bin/python3 newfolder zsh: /usr/local/bin/virtualenv: bad interpreter: /usr/local/opt/python3/bin/python3.6: no such file or directory  Also I have tried running the command with quotes like so: virtualenv --python='/usr/local/bin/python3' newfolder zsh: /usr/local/bin/virtualenv: bad interpreter: /usr/local/opt/python3/bin/python3.6: no such file or directory  Please note I am supplying the correct path to python3 as far as I can tell. Here is what I get when I run which python3 which python3 /usr/local/bin/python3  Also virtualenv appears to be correctly installed. Here is evidence for this: pip3 install virtualenv Requirement already satisfied: virtualenv in /Users/mathewlewis/Library/Python/3.7/lib/python/site-packages (16.7.9)  Also, in case this is relevant, the software I have is currently mac os catalina 10.15.2 Not only would I like a solution (as has been given at this point) I would also like a reason why this didn't work.

Answer: Try:  python3 -m venv venv source ./venv/bin/activate
Question: My request seems unorthodox, but I would like to quickly package an old repository, consisting mostly of python executable scripts. The problem is that those scripts were not designed as modules, so some of them execute code directly at the module top-level, and some other have the if __name__=='__main__' part. How would you distribute those scripts using setuptools, without too much rewrite?  I know I could just put them under the scripts option of setup(), but it's not advised, and also it doesn't allow me to rename them. I would like to skip defining a main() function in all those scripts, also because some scripts call weird recursive functions with side effects on global variables, so I'm a bit afraid of breaking stuff. When I try providing only the module name as console_scripts (e.g "myscript=mypkg.myscript" instead of "myscript=mypkg.myscript:main"), it logically complains after installation that a module is not callable.  Is there a way to create scripts from modules? At least when they have a if __name__=='__main__'?

Answer: There's some design considerations, but I would recommend using a __main__.py for this.  it allows all command line invocations to share argument parsing logic you don't have to touch the scripts, at all it is explicit (no do-nothing functions that exist to trigger import) it enables refactoring out any other common stateful logic, since __main__.py is never imported when you import your package convention. People expect to find command line invocations in __main__.py.  __main__.py from pathlib import Path from runpy import run_path pkg_dir = Path(__file__).resolve().parent def execute_script(): script_pth = pkg_dir / "local path to script" run_path(str(script_pth), run_name="__main__")  Then, you can set your_package.__main__:execute_script as a console script in setup.py/pyproject.toml. You can obviously have as many scripts as you like this way.
Question: When interrupting execution of python async code with PyCharm's interactive debugger (breakpoints) we can inspect the environment with PyCharm's debugging tools like "evaluate expression" or "Execute Line in Python Console". How can we evaluate coroutines within these debugging tools?

Answer: Update from October 2022 Make sure you have the latest version of PyCharm, because they said they added this