eshangj/stackoverflow_q_and_a_sample huggingface dataset:
Question: In the following function definition, what do the * and / account for? def func(self, param1, param2, /, param3, *, param4, param5): print(param1, param2, param3, param4, param5)  NOTE: Not to mistake with the single|double asterisks in *args | **kwargs (solved here)

Answer: The function parameter syntax(/) is to indicate that some function parameters must be specified positionally and cannot be used as keyword arguments.(This is new in Python 3.8) Documentation specifies some of the use cases/benefits of positional-only parameters   It allows pure Python functions to fully emulate behaviors of existing C coded functions. For example, the built-in pow() function does not accept keyword arguments: def pow(x, y, z=None, /): "Emulate the built in pow() function" r = x ** y return r if z is None else r%z   Another use case is to preclude keyword arguments when the parameter name is not helpful. For example, the builtin len() function has the signature len(obj, /). This precludes awkward calls such as: len(obj='hello') # The "obj" keyword argument impairs readability   A further benefit of marking a parameter as positional-only is that it allows the parameter name to be changed in the future without risk of breaking client code. For example, in the statistics module, the parameter name dist may be changed in the future. This was made possible with the following function specification: def quantiles(dist, /, *, n=4, method='exclusive') ...     Where as * is used to force the caller to use named arguments. Django documentation contains a section which clearly explains a usecase of named arguments.  Form fields no longer accept optional arguments as positional arguments To help prevent runtime errors due to incorrect ordering of form field arguments, optional arguments of built-in form fields are no longer accepted as positional arguments. For example: forms.IntegerField(25, 10) raises an exception and should be replaced with: forms.IntegerField(max_value=25, min_value=10)  Suppose we have a method called func: def func(self, param1, param2, /, param3, *, param4, param5): print(param1, param2, param3, param4, param5)  It must called with obj.func(10, 20, 30, param4=50, param5=60)  OR obj.func(10, 20, param3=30, param4=50, param5=60)  ie,  param1, param2 must be specified positionally. param3 can be called either with positional or keyword argument. param4 and param5 must be called with keyword argument.  DEMO: >>> class MyClass(object): ... def func(self, param1, param2, /, param3, *, param4, param5): ... return param1, param2, param3, param4, param5 ... >>> obj = MyClass() >>> >>> assert obj.func(10, 20, 30, param4=40, param5=50), obj.func( ... 10, 20, param3=30, param4=40, param5=50 ... )
Question: I am looking for the best solution for communication between async tasks and methods/functions that run in a thread pool executor from concurrent.futures. In previous synchronous projects, I would use the queue.Queue class. I assume that any method should be thread safe and therefore the asyncio.queue will not work.  I have seen people extend the queue.Queue class to do something like: class async_queue(Queue): async def aput(self, item): self.put_nowait(item) async def aget(self): resp = await asyncio.get_event_loop().run_in_executor( None, self.get ) return resp  Is there a better way? 

Answer: I would recommend going the other way around: using the asyncio.Queue class to communicate between the two worlds. This has the advantage of not having to spend a slot in the thread pool on operations that take a long time to complete, such as a get(). Here is an example: class Queue: def __init__(self): self._loop = asyncio.get_running_loop() self._queue = asyncio.Queue() def sync_put_nowait(self, item): self._loop.call_soon(self._queue.put_nowait, item) def sync_put(self, item): asyncio.run_coroutine_threadsafe(self._queue.put(item), self._loop).result() def sync_get(self): return asyncio.run_coroutine_threadsafe(self._queue.get(item), self._loop).result() def async_put_nowait(self, item): self._queue.put_nowait(item) async def async_put(self, item): await self._queue.put(item) async def async_get(self): return await self._queue.get()  The methods prefixed with sync_ are meant to be invoked by sync code (running outside the event loop thread). The ones prefixed with async_ are to be called by code running in the event loop thread, regardless of whether they are actually coroutines. (put_nowait, for example, is not a coroutine, but it still must be distinguished between a sync and an async version.)
Question: This python code is running on the local computer but is not running on  Digital Ocean Amazon AWS Google Collab Heroku  and many other VPS. It shows different errors at different times. import requests headers = { 'authority': 'beta.nseindia.com', 'cache-control': 'max-age=0', 'dnt': '1', 'upgrade-insecure-requests': '1', 'user-agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/79.0.3945.117 Safari/537.36', 'sec-fetch-user': '?1', 'accept': 'text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.9', 'sec-fetch-site': 'none', 'sec-fetch-mode': 'navigate', 'accept-encoding': 'gzip, deflate, br', 'accept-language': 'en-US,en;q=0.9,hi;q=0.8', } params = ( ('symbol', 'BANKNIFTY'), ) response = requests.get('https://nseindia.com/api/quote-derivative', headers=headers, params=params) #NB. Original query string below. It seems impossible to parse and #reproduce query strings 100% accurately so the one below is given #in case the reproduced version is not "correct". # response = requests.get('https://nseindia.com/api/quote-derivative?symbol=BANKNIFTY', headers=headers)  Is there any mistake in the above code? What I am missing? I copied the header data from Chrome Developer Tools> Network in incognito mode used https://curl.trillworks.com/ site to generate the python code from the curl command. But the curl command is working fine and giving fine output- curl "https://nseindia.com/api/quote-derivative?symbol=BANKNIFTY" -H "authority: beta.nseindia.com" -H "cache-control: max-age=0" -H "dnt: 1" -H "upgrade-insecure-requests: 1" -H "user-agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/79.0.3945.117 Safari/537.36" -H "sec-fetch-user: ?1" -H "accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.9" -H "sec-fetch-site: none" -H "sec-fetch-mode: navigate" -H "accept-encoding: gzip, deflate, br" -H "accept-language: en-US,en;q=0.9,hi;q=0.8" --compressed  How come the curl command is working but the python generated out of the curl command is not?

Answer: Use the nsefetch() function as documented here https://unofficed.com/nse-python/documentation/nsefetch/ In case you want python-requests method from nsepython import * payload= nsefetch('https://www.nseindia.com/live_market/dynaContent/live_watch/option_chain/optionKeys.jsp?segmentLink=17&instrument=OPTIDX&symbol=BANKNIFTY') print(payload)  This will work in all desktops and laptops. In case you want curl method from nsepythonserver import * payload= nsefetch('https://www.nseindia.com/live_market/dynaContent/live_watch/option_chain/optionKeys.jsp?segmentLink=17&instrument=OPTIDX&symbol=BANKNIFTY') print(payload)  This will work in all servers. Like Linux servers.
Question: I have the following code: async some_callback(args): await some_function()  and I need to give it to a Thread as a target: _thread = threading.Thread(target=some_callback, args=("some text")) _thread.start()  The error that I get is "some_callback is never awaited". Any ideas how can I solve this problem?

Answer: You can do it by adding function between to execute async: import asyncio async def some_callback(args): await some_function() def between_callback(args): loop = asyncio.new_event_loop() asyncio.set_event_loop(loop) loop.run_until_complete(some_callback(args)) loop.close() _thread = threading.Thread(target=between_callback, args=("some text")) _thread.start()
Question: I'd like to read a big 3GB .mbox file coming from a Gmail backup. This works: import mailbox mbox = mailbox.mbox(r"D:\All mail Including Spam and Trash.mbox") for i, message in enumerate(mbox): print("from :",message['from']) print("subject:",message['subject']) if message.is_multipart(): content = ''.join(part.get_payload(decode=True) for part in message.get_payload()) else: content = message.get_payload(decode=True) print("content:",content) print("**************************************") if i == 10: break  except it takes more than 40 seconds for the first 10 messages only. Is there a faster way to access to a big .mbox file with Python?

Answer: Here's a quick and dirty attempt to implement a generator to read in an mbox file message by message. I have opted to simply ditch the information from the From  separator; I'm guessing maybe the real mailbox library might provide more information, and of course, this only supports reading, not searching or writing back to the input file. #!/usr/bin/env python3 import email from email.policy import default class MboxReader: def __init__(self, filename): self.handle = open(filename, 'rb') assert self.handle.readline().startswith(b'From ') def __enter__(self): return self def __exit__(self, exc_type, exc_value, exc_traceback): self.handle.close() def __iter__(self): return iter(self.__next__()) def __next__(self): lines = [] while True: line = self.handle.readline() if line == b'' or line.startswith(b'From '): yield email.message_from_bytes(b''.join(lines), policy=default) if line == b'': break lines = [] continue lines.append(line)  Usage: with MboxReader(mboxfilename) as mbox: for message in mbox: print(message.as_string())  The policy=default argument (or any policy instead of default if you prefer, of course) selects the modern EmailMessage library which was introduced in Python 3.3 and became official in 3.6. If you need to support older Python versions from before America lost its mind and put an evil clown in the White House simpler times, you will want to omit it; but really, the new API is better in many ways.
Question: I've seen several examples of .dockerignore files for Python projects where *.pyc files and/or __pycache__ folders are ignored: **/__pycache__ *.pyc  Since these files/folders are going to be recreated in the container anyway, I wonder if it's a good practice to do so.

Answer: Yes, it's a recommended practice. There are several reasons: Reduce the size of the resulting image In .dockerignore you specify files that won't go to the resulting image, it may be crucial when you're building the smallest image. Roughly speaking the size of bytecode files is equal to the size of actual files. Bytecode files aren't intended for distribution, that's why we usually put them into .gitignore as well.  Cache related problems In earlier versions of Python 3.x there were several cached related issues:  Python’s scheme for caching bytecode in .pyc files did not work well in environments with multiple Python interpreters. If one interpreter encountered a cached file created by another interpreter, it would recompile the source and overwrite the cached file, thus losing the benefits of caching.  Since Python 3.2 all the cached files prefixed with interpreter version as mymodule.cpython-32.pyc and presented under __pychache__ directory. By the way, starting from Python 3.8 you can even control a directory where the cache will be stored. It may be useful when you're restricting write access to the directory but still want to get benefits of cache usage. Usually, the cache system works perfectly, but someday something may go wrong. It worth to note that the cached .pyc (lives in the same directory) file will be used instead of the .py file if the .py the file is missing. In practice, it's not a common occurrence, but if some stuff keeps up being "there", thinking about remove cache files is a good point. It may be important when you're experimenting with the cache system in Python or executing scripts in different environments.  Security reasons Most likely that you don't even need to think about it, but cache files can contain some sort of sensitive information. Due to the current implementation, in .pyc files presented an absolute path to the actual files. There are situations when you don't want to share such information.  It seems that interacting with bytecode files is a quite frequent necessity, for example, django-extensions have appropriate options compile_pyc and clean_pyc.
Question: I wrote this script to save a file from an SFTP remote folder to a local folder. It then removes the file from the SFTP. I want to change it so it stops removing files and instead saves them to a backup folder on the SFTP. How do I do that in pysftp? I cant find any documentation regarding it... import pysftp cnopts = pysftp.CnOpts() cnopts.hostkeys = None myHostname = "123" myUsername = "456" myPassword = "789" with pysftp.Connection(host=myHostname, username=myUsername, password="789", cnopts=cnopts) as sftp: sftp.cwd("/Production/In/") directory_structure = sftp.listdir_attr() local_dir= "D:/" remote_dir = "/Production/" remote_backup_dir = "/Production/Backup/" for attr in directory_structure: if attr.filename.endswith(".xml"): file = attr.filename sftp.get(remote_dir + file, local_dir + file) print("Moved " + file + " to " + local_dir) sftp.remove(remote_dir + file)  Don't worry about my no hostkey or the password in plain. I'm not keeping it that way once i get the script working :)

Answer: Use Connection.rename: sftp.rename(remote_dir + file, remote_backup_dir + file)   Obligatory warnings:  Do not set cnopts.hostkeys = None, unless you do not care about security. For the correct solution see Verify host key with pysftp. Do not use pysftp. It's dead. Use Paramiko. See pysftp vs. Paramiko.
Question: I'm little confused about how does the class StratifiedShuffleSplit of Sklearn works. The code below is from Géron's book "Hands On Machine Learning", chapter 2, where he does a stratified sampling. from sklearn.model_selection import StratifiedShuffleSplit split = StratifiedShuffleSplit(n_splits=1, test_size=0.2, random_state=42) for train_index, test_index in split.split(housing, housing["income_cat"]): strat_train_set = housing.loc[train_index] strat_test_set = housing.loc[test_index]  Especially, what is been doing in split.split? Thanks!

Answer: Since you did not provide a dataset, I use sklearn sample to answer this question. Prepare dataset # generate data import numpy as np from sklearn.model_selection import StratifiedShuffleSplit data = np.array([[1, 2], [3, 4], [1, 2], [3, 4], [1, 2], [3, 4]]) group_label = np.array([0, 0, 0, 1, 1, 1])  This generate a dataset data, which has 6 obseravations and 2 variables. group_label has 2 value, means group 0 and group 1. In this case, group 0 contains 3 samples, same is group 1. To be general, the group size are not need to be the same. Create a StratifiedShuffleSplit object instance sss = StratifiedShuffleSplit(n_splits=5, test_size=0.5, random_state=0) sss.get_n_splits(data, group_label)  Out:  5  In this step, you can create a instance of StratifiedShuffleSplit, you can tell the function how to split(At random_state = 0,split data 5 times,each time 50% of data will split to test set). However, it only split data when you call it in the next step. Call the instance, and split data. # the instance is actually a generater type(sss.split(data, group_label)) # split data for train_index, test_index in sss.split(data, group_label): print("n_split",,"TRAIN:", train_index, "TEST:", test_index) X_train, X_test = X[train_index], X[test_index] y_train, y_test = y[train_index], y[test_index]  out: TRAIN: [5 2 3] TEST: [4 1 0] TRAIN: [5 1 4] TEST: [0 2 3] TRAIN: [5 0 2] TEST: [4 3 1] TRAIN: [4 1 0] TEST: [2 3 5] TRAIN: [0 5 1] TEST: [3 4 2]  In this step, spliter you defined in the last step will generate 5 split of data one by one. For instance, in the first split, the original data is shuffled and sample 5,2,3 is selected as train set, this is also a stratified sampling by group_label; in the second split, the data is shuffled again and sample 5,1,4 is selected as train set; etc..
Question: I am writing a telegram to the bot. I ran into such a problem. I need the bot to send a message (text) when clicked on which it was copied (as a token from @BotFather)

Answer: If I understand you correctly, you wish to send a message, that when the user presses it, the text is automatically copied to the user's clipboard, just like the BotFather sends the API token?   This is done by the MarkDown parse_mode; Send a message with &parse_mode=MarkDown and wrap the 'pressable' text in back-ticks (`): Hi. `Press me!`!  https://api.telegram.org/bot<token>/sendMessage?chat_id=<id>&text=Hi! `Press me!`&parse_mode=MarkDown    EDIT: Bases on OP's comment you're looking for a python-telegram-bot solution. From there documentation; bot.send_message( chat_id=chat_id, text="*bold* _italic_ `fixed width font` [link](http://google.com).", parse_mode=telegram.ParseMode.MARKDOWN )
Question: Specifically this popup box:  It appears almost every time I type anything, and it's kind of getting in the way.  I have disabled code completion in the settings, and uninstalled Kite, and disabled Jedi. Any ideas?

Answer: (Spyder maintainer here) To disable those completions (called fallback completions) in Spyder 5, you need to go to Tools > Preferences > Completion and Linting > General and deactivate the option called Enable Fallback provider. For Spyder 4, please go to Tools > Preferences > Completion and Linting > Advanced and deactivate the option called Enable fallback completions.
Question: I'm working with Flask-Marshmallow for validating request and response schemas in Flask app. I was able to do simple validations for request.form and request.args when there are simple fields like Int, Str, Float etc.  I have a case where I need to upload a file using a form field - file_field. It should contain the file content. How can I validate if this field is present or not and what is the format of file etc. Is there any such field in Marshmallow that I can use like fields.Int() or fields.Str() I have gone through the documentation here but haven't found any such field.

Answer: You can use fields.Raw: import marshmallow class CustomSchema(marshmallow.Schema): file = marshmallow.fields.Raw(type='file')  If you are using Swagger, you would then see something like this:  Then in your view you can access the file content with flask.request.files. For a full example and more advanced topics, check out my project.
Question: def custom_asymmetric_train(y_true, y_pred): residual = (y_true - y_pred).astype("float") grad = np.where(residual>0, -2*10.0*residual, -2*residual) hess = np.where(residual>0, 2*10.0, 2.0) return grad, hess  I want to write this statement:   case when residual>=0 and residual<=0.5 then -2*1.2*residual when residual>=0.5 and residual<=0.7 then -2*1.*residual when residual>0.7 then -2*2*residual end )  however np.where cannot write &(and) logic . How do I write this case when logic in the np.where in python.  Thanks 

Answer: This statement can be written using np.select as: import numpy as np residual = np.random.rand(10) -0.3 # -0.3 to get some negative values condlist = [(residual>=0.0)&(residual<=0.5), (residual>=0.5)&(residual<=0.7), residual>0.7] choicelist = [-2*1.2*residual, -2*1.0*residual,-2*2.0*residual] residual = np.select(condlist, choicelist, default=residual)  Note that, when multiple conditions are satisfied in condlist, the first one encountered is used. When all conditions evaluate to False, it will use the default value. Moreover, for your information, you need to use bitwise operator & on boolean numpy arrays as and python keyword won't work on them. Let's benchmark these answers: residual = np.random.rand(10000) -0.3 def charl_3where(residual): residual = np.where((residual>=0.0)&(residual<=0.5), -2*1.2*residual, residual) residual = np.where((residual>=0.5)&(residual<=0.7), -2*1.0*residual, residual) residual = np.where(residual>0.7, -2*2.0*residual, residual) return residual def yaco_select(residual): condlist = [(residual>=0.0)&(residual<=0.5), (residual>=0.5)&(residual<=0.7), residual>0.7] choicelist = [-2*1.2*residual, -2*1.0*residual,-2*2.0*residual] residual = np.select(condlist, choicelist, default=residual) return residual %timeit charl_3where(residual) >>> 112 µs ± 1.7 µs per loop (mean ± std. dev. of 7 runs, 10000 loops each) %timeit yaco_select(residual) >>> 141 µs ± 2 µs per loop (mean ± std. dev. of 7 runs, 10000 loops each)  let's try to optimize these with numba from numba import jit @jit(nopython=True) def yaco_numba(residual): out = np.empty_like(residual) for i in range(residual.shape[0]): if residual[i]<0.0 : out[i] = residual[i] elif residual[i]<=0.5 : out[i] = -2*1.2*residual[i] elif residual[i]<=0.7: out[i] = -2*1.0*residual[i] else: # residual>0.7 out[i] = -2*2.0*residual[i] return out %timeit yaco_numba(residual) >>> 6.65 µs ± 123 ns per loop (mean ± std. dev. of 7 runs, 100000 loops each)  Final check res1 = charl_3where(residual) res2 = yaco_select(residual) res3 = yaco_numba(residual) np.allclose(res1,res3) >>> True np.allclose(res2,res3) >>> True  This one is about 15x faster than the previously best one. Hope this helps.
Question: import matplotlib.pyplot as plt import numpy as np randomnums = np.random.normal(loc=9,scale=6, size=400).astype(int)+15  Output: array([25, 22, 19, 26, 24, 9, 19, 32, 30, 25, 29, 17, 21, 14, 17, 27, 27, 28, 17, 17, 20, 21, 16, 28, 20, 24, 15, 20, 20, 13, 33, 21, 30, 27, 8, 22, 24, 25, 23, 13, 24, 20, 16, 32, 15, 26, 34, 16, 21, 21, 28, 22, 23, 18, 20, 22, 23, 22, 23, 26, 22, 25, 19, 29, 14, 27, 21, 23, 24, 19, 25, 15, 22, 23, 19, 19, 23, 21, 22, 17, 25, 15, 24, 25, 23 ...  h = sorted(randomnums) plt.hist(h,density=False) plt.show()  Output:  From my research I found only how to plot numbers on top of a bar chart, but what I want is to plot on top of a histogram chart. Is it possible?

Answer: An adapted version of the answer I linked in the comments of the question. Thanks a lot for the suggestions in the comments below this post! import matplotlib.pyplot as plt import numpy as np h = np.random.normal(loc=9,scale=6, size=400).astype(int)+15 fig, ax = plt.subplots(figsize=(16, 10)) ax.hist(h, density=False) for rect in ax.patches: height = rect.get_height() ax.annotate(f'{int(height)}', xy=(rect.get_x()+rect.get_width()/2, height), xytext=(0, 5), textcoords='offset points', ha='center', va='bottom')  ...gives e.g.  See also: matplotlib.axes.Axes.annotate.