eshangj/stackoverflow_q_and_a_sample huggingface dataset:
Question: I am working a sample program that reads from a datasource (csv or rdbms) in chunks, makes some transformation and sends it via socket to a server. But because the csv is very large, for testing purpose I want to break the reading after few chunks. Unfortunately something goes wrong and I do not know what and how to fix it. Probably I have to do some cancellation, but now sure where and how. I get the following error: Task was destroyed but it is pending! task: <Task pending coro=<<async_generator_athrow without __name__>()>>  The sample code is: import asyncio import json async def readChunks(): # this is basically a dummy alternative for reading csv in chunks df = [{"chunk_" + str(x) : [r for r in range(10)]} for x in range(10)] for chunk in df: await asyncio.sleep(0.001) yield chunk async def send(row): j = json.dumps(row) print(f"to be sent: {j}") await asyncio.sleep(0.001) async def main(): i = 0 async for chunk in readChunks(): for k, v in chunk.items(): await asyncio.gather(send({k:v})) i += 1 if i > 5: break #print(f"item in main via async generator is {chunk}") loop = asyncio.get_event_loop() loop.run_until_complete(main()) loop.close() 

Answer: Many async resources, such as generators, need to be cleaned up with the help of an event loop. When an async for loop stops iterating an async generator via break, the generator is cleaned up by the garbage collector only. This means the task is pending (waits for the event loop) but gets destroyed (by the garbage collector). The most straightforward fix is to aclose the generator explicitly: async def main(): i = 0 aiter = readChunks() # name iterator in order to ... try: async for chunk in aiter: ... i += 1 if i > 5: break finally: await aiter.aclose() # ... clean it up when done   These patterns can be simplified using the asyncstdlib (disclaimer: I maintain this library). asyncstdlib.islice allows to take a fixed number of items before cleanly closing the generator: import asyncstdlib as a async def main(): async for chunk in a.islice(readChunks(), 5): ...  If the break condition is dynamic, scoping the iterator guarantees cleanup in any case: import asyncstdlib as a async def main(): async with a.scoped_iter(readChunks()) as aiter: async for idx, chunk in a.enumerate(aiter): ... if idx >= 5: break
Question: Coming from primarily coding in Java and wanted to know if Python could use conditionals and different kinds of incrementing inside its for loops like Java and C can. Sorry if this seems like a simple question. i.e.: boolean flag = True for(int i = 1; i < 20 && flag; i *= 2) { //Code in here } 

Answer: Not directly. A for loop iterates over a pre-generated sequence, rather than generating the sequence itself. The naive translation would probably look something like flag = True i = 1 while i < 20: if not flag: break ... if some_condition: flag = False i *= 2  However, your code probably could execute the break statement wherever you set flag to False, so you could probably get rid of the flag altogether. i = 1 while i < 20: ... if some_condition: break i *= 2  Finally, you can define your own generator to iterate over def powers_of_two(): i = 1 while True: yield i i *= 2 for i in powers_of_two(): ... if some_condition: break
Question: Suppose I have a dataframe containing a column of probability. Now I create a map function which returns 1 if the probability is greater than a threshold value, otherwise returns 0. Now the catch is that I want to specify the threshold by giving it as an argument to the function, and then mapping it on the pandas dataframe. Take the code example below: def partition(x,threshold): if x<threshold: return 0 else: return 1 df = pd.DataFrame({'probability':[0.2,0.8,0.4,0.95]}) df2 = df.map(partition)  My question is, how would the last line work, i.e. how do I pass the threshold value inside my map function?

Answer: We can use Dataframe.applymap df2 = df.applymap(lambda x: partition(x, threshold=0.5))  Or if only one column: df['probability']=df['probability'].apply(lambda x: partition(x, threshold=0.5))  but it is not neccesary here. You can do: df2 = df.ge(threshold).astype(int)  I recommend you see it
Question: I'm currently rewriting a bunch of matlab code into julia. These codes envolves a lot of math and, particularly, interpolation functions for a 3D mesh. It is easy to deal with this in matlab: all I need to do is to use interp3 function. Once I coundn't find any simple way to do similar in Julia, I'm trying to use some Scipy features through PyCall. Now, the problem: I've already installed PyCall, changed ENV[PYTHON] to the path of my own installed anaconda. No metter what, and I extensively looked for solutions, I still get the following error message: julia> pyimport("scipy") ERROR: PyError (PyImport_ImportModule The Python package scipy could not be found by pyimport. Usually this means that you did not install scipy in the Python version being used by PyCall. PyCall is currently configured to use the Python version at: /usr/bin/python3 and you should use whatever mechanism you usually use (apt-get, pip, conda, etcetera) to install the Python package containing the scipy module. One alternative is to re-configure PyCall to use a different Python version on your system: set ENV["PYTHON"] to the path/name of the python executable you want to use, run Pkg.build("PyCall"), and re-launch Julia. Another alternative is to configure PyCall to use a Julia-specific Python distribution via the Conda.jl package (which installs a private Anaconda Python distribution), which has the advantage that packages can be installed and kept up-to-date via Julia. As explained in the PyCall documentation, set ENV["PYTHON"]="", run Pkg.build("PyCall"), and re-launch Julia. Then, To install the scipy module, you can use `pyimport_conda("scipy", PKG)`, where PKG is the Anaconda package the contains the module scipy, or alternatively you can use the Conda package directly (via `using Conda` followed by `Conda.add` etcetera). ) <class 'ModuleNotFoundError'> ModuleNotFoundError("No module named 'scipy'",) Stacktrace: [1] pyimport(::String) at /home/gabriel/.julia/packages/PyCall/zqDXB/src/PyCall.jl:536 [2] top-level scope at none:0  Also, everything I tried, I tried both on windows 10 and Linux. I don't know what to do anymore! I would much appreciate the help! Thanks in advance!

Answer: Install scipy with Conda - Julia's interface to Python's packages. using Conda Conda.add("scipy")  now pyimport("scipy") will work like charm. Note that with a custom Python installation various things can happen (and you are left on your own with managing that), hence I recommend you to use Python built-into Julia. This is how you switch back to in-built Python: using Pkg ENV["PYTHON"]="" Pkg.build("PyCall")
Question: I am using the threading module to host a web server and a Discord bot at the same time. Everything runs fine on Windows but as soon as I load it onto my Linux server I get the following error: Starting Bot Exception in thread Bot: Traceback (most recent call last): File "/usr/lib/python3.8/asyncio/unix_events.py", line 95, in add_signal_handler signal.set_wakeup_fd(self._csock.fileno()) ValueError: set_wakeup_fd only works in main thread During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/usr/lib/python3.8/threading.py", line 932, in _bootstrap_inner self.run() File "/home/webadmin/discordbot/bot/moduls/m_threadingmaker.py", line 15, in run self.client.run(self.args[0]) File "/home/webadmin/discordbot/bot/venv/lib/python3.8/site-packages/discord/client.py", line 614, in run loop.add_signal_handler(signal.SIGINT, lambda: loop.stop()) File "/usr/lib/python3.8/asyncio/unix_events.py", line 97, in add_signal_handler raise RuntimeError(str(exc)) RuntimeError: set_wakeup_fd only works in main thread  I have upgraded from python 3.7 up to python 3.8 but I still have the same error. Here is my code: main.py (webserver worked) dcbot = m_threadingmaker.myThread("Bot", client, secrets.token) webserver = m_threadingmaker.myThread("Flask", app, 'localhost', '7010') #webserver.start() dcbot.start()  M_threadingmaker.py from threading import Thread class myThread (Thread): def __init__(self, name, client, *args): Thread.__init__(self) self.name = name self.client = client self.args = args def run(self): print("Starting " + self.name) if self.name == "Flask": self.client.run(host=self.args[0], port=self.args[1]) else: self.client.run(self.args[0]) print("Exiting " + self.name) 

Answer: I'd suggest you to use client.start() in async coroutine instead of client.run() in separate thread. More detailed example here
Question: I have a tuple- ('name@mail.com',). I want to unpack it to get 'name@mail.com'. How can I do so? I am new to Python so please excuse.

Answer: tu = ('name@mail.com',) str = tu[0] print(str) #will return 'name@mail.com'  A tuple is a sequence type, which means the elements can be accessed by their indices.
Question: The gzip docs for Python 3 states that  Calling a GzipFile object’s close() method does not close fileobj, since you might wish to append more material after the compressed data  Does this mean that the gzip file handler f_in is not closed if we do the following import gzip import shutil with gzip.open('/home/joe/file.txt.gz', 'rb') as f_in: with open('/home/joe/file.txt', 'wb') as f_out: shutil.copyfileobj(f_in, f_out)  If so, will this cause a leak if this code is executed multiple times?

Answer: The warning about fileobj not being closed only applies when you open the file, and pass it to the GzipFile via the fileobj= parameter. When you pass only a filename, GzipFile "owns" the file handle and will also close it.
Question: I got a dataframe which looks like below: df: Time of Day Season value Day Shoulder 30.581606 Day Summer 25.865560 Day Winter 42.644530 Evening Shoulder 39.954759 Evening Summer 32.053458 Evening Winter 53.678297 Morning Shoulder 32.171245 Morning Summer 25.070815 Morning Winter 42.876667 Night Shoulder 22.082042 Night Summer 17.510290 Night Winter 33.262356  I am plotting the values in line plot using seaborn using following code: g = sns.lineplot(x='Time of Day',y='value',data=df,hue='Season')  it generates the following graph:  The problem with the graph is that order of the x-axis is not in the desired order. I want to change the order of a-axis value to ['Morning', 'Day','Evening','Night']. I tried to change it with the following command: g.set_xticklabels(['Morning','Day','Evening','Night'])  But this command only changes the label of the x-axis, not the order of data points. Could anyone help me in fixing the issue?

Answer:  Use pandas.Categorical to set the categorical order of 'Time of Day' in the df. Tested in python 3.11, pandas 1.5.3, matplotlib 3.7.1, seaborn 0.12.2  import pandas as pd import matplotlib.pyplot as plt import seaborn as sns data = {'Time of Day': ['Day', 'Day', 'Day', 'Evening', 'Evening', 'Evening', 'Morning', 'Morning', 'Morning', 'Night', 'Night', 'Night'], 'Season': ['Shoulder', 'Summer', 'Winter', 'Shoulder', 'Summer', 'Winter', 'Shoulder', 'Summer', 'Winter', 'Shoulder', 'Summer', 'Winter'], 'value': [30.581606, 25.865560000000002, 42.644529999999996, 39.954759, 32.053458, 53.678297, 32.171245, 25.070815, 42.876667, 22.082042, 17.510289999999998, 33.262356]} # create dataframe df = pd.DataFrame(data) # set categorical order df['Time of Day'] = pd.Categorical(df['Time of Day'], categories=['Morning', 'Day', 'Evening', 'Night'], ordered=True) # plot ax = sns.lineplot(x='Time of Day', y='value', data=df, hue='Season') sns.move_legend(ax, bbox_to_anchor=(1, 0.5), loc='center left', frameon=False)     If you are plotting data against a categorical x-axis, then consider using a categorical plot like sns.pointplot, where the x-axis can be ordered with the order parameter.  The figure-level plot, sns.catplot with kind='point' also works.    order=['Morning', 'Day', 'Evening', 'Night'] ax = sns.pointplot(x='Time of Day', y='value', data=df, hue='Season', order=order, marker='') sns.move_legend(ax, bbox_to_anchor=(1, 0.5), loc='center left', frameon=False)
Question: I'm in the process of converting my projects to use flit as their build backend using pyroject.toml as defined in PEP517. I still have some projects that will continue to use setuptools as their build backend. Some projects may not be PEP 517 compliant and will use the legacy setup.py build system. I'm using the latest pip (20.1) as the build frontend, regardless of whether the build backend for a particular project is setuptools or flit. However, I can't figure out how to generate an sdist with pip. I can generate a wheel with pip, regardless of the build backend $ pip wheel . -w dist --no-deps  But there doesn't appear to be a similar pip command to generate an sdist, even though all build backends must support creating sdists, as defined in PEP517. Shouldn't all build front-ends support building sdist's? Should I be using another tool? I'd like to use the same command to build all of my projects regardless of what their build backend is.

Answer: The Python Packaging Guide recommends building packages using the new build library maintained by the same Python Packaging Authority that maintains pip. https://github.com/pypa/build python -m build . --sdist
Question: I have an function called "value" that makes heavy calculation... The result of the function is always the same if the dataset is not changed for the identifier. Once the dataset is changed for some identifier, I want to clear the cache, and let the function calculate it again. You can better understand me by looking at this code: from functools import cached_property class Test: identifiers = {} dataset = an empty object of dataset type def __init__(self, identifier, ...) self.identifier = identifier ... Test.identifiers[identifier] = self ... @cached_property def value(self): result = None # heavy calculate based on dataset return result @classmethod def get(cls, identifier): if identifier in cls.identifiers: return cls.identifiers[identifier] else: return cls(identifier, ...) @classmethod def update(cls, dataset): for block in dataset: # assume there is block['identifier'] in each block # here i want to clear the cache of value() function instance = cls.get(block['identifier']) # clear @cached_property of instance cls.dataset.append(block) 

Answer: As you can read in the CPython source, the value for a cached_property in Python 3.8 is stored in an instance variable of the same name. This is not documented, so it may be an implementation detail that you should not rely upon. But if you just want to get it done without regards to compatibility, you can remove the cache with del instance.value. As of Python 3.9, this is documented.
Question: I want to overwrite an existing sheet in an excel file with Pandas dataframe but don't want any changes in other sheets of the same file. How this can be achieved. I tried below code but instead of overwriting, it is appending the data in 'Sheet2'. import pandas as pd from openpyxl import load_workbook book = load_workbook('sample.xlsx') writer = pd.ExcelWriter('sample.xlsx', engine = 'openpyxl') writer.book = book writer.sheets = dict((ws.title, ws) for ws in book.worksheets) df.to_excel(writer, 'sheet2', index = False) writer.save() 

Answer: I didn't find any other option other than this, this would be a quick solution for you. I believe still there's no direct way to do this, correct me if I'm wrong. That's the reason we need to play with these logical ways. import pandas as pd def write_excel(filename,sheetname,dataframe): with pd.ExcelWriter(filename, engine='openpyxl', mode='a') as writer: workBook = writer.book try: workBook.remove(workBook[sheetname]) except: print("Worksheet does not exist") finally: dataframe.to_excel(writer, sheet_name=sheetname,index=False) writer.save() df = pd.DataFrame({'Col1':[1,2,3,4,5,6], 'col2':['foo','bar','foobar','barfoo','foofoo','barbar']}) write_excel('PRODUCT.xlsx','PRODUCTS',df)  Let me know if you found this helpful, or ignore it if you need any other better solution.
Question: What would be the suggested way to run something like the following in python: self.cursor.execute('SET FOREIGN_KEY_CHECKS=0; DROP TABLE IF EXISTS %s; SET FOREIGN_KEY_CHECKS=1' % (table_name,))  For example, should this be three separate self.cursor.execute(...) statements? Is there a specific method that should be used other than cursor.execute(...) to do something like this, or what is the suggested practice for doing this? Currently the code I have is as follows: self.cursor.execute('SET FOREIGN_KEY_CHECKS=0;') self.cursor.execute('DROP TABLE IF EXISTS %s;' % (table_name,)) self.cursor.execute('SET FOREIGN_KEY_CHECKS=1;') self.cursor.execute('CREATE TABLE %s select * from mytable;' % (table_name,))  As you can see, everything is run separately...so I'm not sure if this is a good idea or not (or rather -- what the best way to do the above is). Perhaps BEGIN...END ?

Answer: I would create a stored procedure: DROP PROCEDURE IF EXISTS CopyTable; DELIMITER $$ CREATE PROCEDURE CopyTable(IN _mytable VARCHAR(64), _table_name VARCHAR(64)) BEGIN SET FOREIGN_KEY_CHECKS=0; SET @stmt = CONCAT('DROP TABLE IF EXISTS ',_table_name); PREPARE stmt1 FROM @stmt; EXECUTE stmt1; SET FOREIGN_KEY_CHECKS=1; SET @stmt = CONCAT('CREATE TABLE ',_table_name,' as select * from ', _mytable); PREPARE stmt1 FROM @stmt; EXECUTE stmt1; DEALLOCATE PREPARE stmt1; END$$ DELIMITER ;  and then just run: args = ['mytable', 'table_name'] cursor.callproc('CopyTable', args)  keeping it simple and modular. Of course you should do some kind of error checking and you could even have the stored procedure return a code to indicate success or failure.
Question: This traceback mess up all my program and I still cant fix it I have tried all methods and it didn't help! Here's the problem:  ffi_prep_closure(): bad user_data (it seems that the version of the libffi library seen at runtime is different from the 'ffi.h' file seen at compile-time) 

Answer: It is cffi python package issue. Try to download the source package tar.gz from https://pypi.org/project/cffi/#files and install it manually using: python setup.py install
Question: What the difference between att_mask and key_padding_mask in MultiHeadAttnetion of pytorch:  key_padding_mask – if provided, specified padding elements in the key will be ignored by the attention. When given a binary mask and a value is True, the corresponding value on the attention layer will be ignored. When given a byte mask and a value is non-zero, the corresponding value on the attention layer will be ignored   attn_mask – 2D or 3D mask that prevents attention to certain positions. A 2D mask will be broadcasted for all the batches while a 3D mask allows to specify a different mask for the entries of each batch.  Thanks in advance.

Answer: The key_padding_mask is used to mask out positions that are padding, i.e., after the end of the input sequence. This is always specific to the input batch and depends on how long are the sequence in the batch compared to the longest one. It is a 2D tensor of shape batch size × input length. On the other hand, attn_mask says what key-value pairs are valid. In a Transformer decoder, a triangle mask is used to simulate the inference time and prevent the attending to the "future" positions. This is what att_mask is usually used for. If it is a 2D tensor, the shape is input length × input length. You can also have a mask that is specific to every item in a batch. In that case, you can use a 3D tensor of shape (batch size × num heads) × input length × input length. (So, in theory, you can simulate key_padding_mask with a 3D att_mask.)
Question: I have seen at many places that using thread local storage to store any data in Django application is not a good practice. But this is the only way I could store my request object. I need to store it because my application has a complex structure. And I can't keep on passing the request object at each function call or class intialization. I need the cookies and headers from my request object, to be passed to some api calls I'm making at different places in the application. I'm using this for reference: https://blndxp.wordpress.com/2016/03/04/django-get-current-user-anywhere-in-your-code-using-a-middleware/ So I'm using a middleware, as mentioned in the reference. And, this is how request is stored from threading import local _thread_locals = local() _thread_locals.request = request  And, this is how data is fetched: getattr(_thread_locals, "request", None)  So does are the data stored in the threads local to that particular request ? Or if another request takes place at the same time, does both of them use the same data ?(Which is certainly not what i want) Or is there any new way of dealing with this old problem(storing request object globally) Note: I'm also using async at places in my Django application(If that matters).

Answer: Yes, using thread-local storage in Django is safe. Django uses one thread to handle each request. Django also uses thread-local data itself, for instance for storing the currently activated locale. While appservers such as Gunicorn and uwsgi can be configured to utilize multiple threads, each request will still be handled by a single thread. However, there have been conflicting opinions on whether using thread-locals is an elegant and well-designed solution. The reasons against using thread-locals boil down to the same reasons why global variables are considered bad practice. This answer discusses a number of them. Still, storing the request object in thread-local data has become a widely used pattern in the Django community. There is even an app Django-CRUM that contains a CurrentRequestUserMiddleware class and the functions get_current_user() and get_current_request(). Note that as of version 3.0, Django has started to implement asynchronous support. I'm not sure what its implications are for apps like Django-CRUM. For the foreseeable future, however, thread-locals can safely be used with Django.
Question: I'm trying to run this project locally but when i try manage.py makemigrations i keep getting the following error: ImportError: cannot import name 'config' from 'decouple'  Here are my steps:  Clone the repository from github Create a virtual environment Install the dependencies  I made some research but i found nothing about what could be generating that error. Can anyone help me out on this? Thanks in advance!I'm running Django 3.

Answer: You might have decouple installed in additional to python-decouple (two different packages). If that is the case simply uninstall decouple pip uninstall decouple  And ensure you have python-decouple installed pip install python-decouple