eshangj/stackoverflow_q_and_a_sample huggingface dataset:
Question: I was implementing asynchronous MySQL query execution using python3.8's inbuilt asyncio package and an installed aiomysql package. Even though I have closed properly all the open cursor and connection, an error message keep on appearing on my console as follows. An open stream object is being garbage collected; call "stream.close()" explicitly.  A summary of the code is given below... #db.py import asyncio class AsyncMysqlSession: def __init__(self, loop, db_settings=DEFAULTDB): self.db_settings = db_settings self.loop = loop async def __aenter__(self): self.conn = await aiomysql.connect(host=self.db_settings['HOST'], port=self.db_settings['PORT'], user=self.db_settings['USER'], password=self.db_settings['PASSWORD'], db=self.db_settings['NAME'], loop=self.loop) self.cursor = await self.conn.cursor(aiomysql.cursors.DictCursor) return self async def __aexit__(self, exception, value, traceback): await self.cursor.close() self.conn.close() async def query(self, sql, *args): await self.cursor.execute(sql, values) await self.conn.commit() rows = await self.cursor.fetchall() return list(rows) async def aiomysql_query(sql, *args): """ Mysql asynchronous connection wrapper """ loop = asyncio.get_event_loop() async with AsyncMysqlSession(loop) as mysql: db_result = await mysql.query(sql, *args) return db_result  aiomysql_query is imported in another file #views.py import asyncio ..... async def main(): ..... ..... await aiomysql_query(sql1, *args1) await aiomysql_query(sql2, *args2) ..... asyncio.run(main()) ....  Am I doing something wrong here (?) or is it improperly shows the error message?. Any lead to resolve this issue will be appreciated... TIA!!

Answer: It seems like you may have just forgotten to close the event loop—in addition to await conn.wait_closed(), which @VPfB advised above. You must close the event loop when manually using lower level method calls such as asyncio.get_event_loop(). Specifically, self.loop.close() must be called. #db.py import asyncio class AsyncMysqlSession: def __init__(self, loop, db_settings=DEFAULTDB): self.db_settings = db_settings self.loop = loop async def __aenter__(self): self.conn = await aiomysql.connect(host=self.db_settings['HOST'], port=self.db_settings['PORT'], user=self.db_settings['USER'], password=self.db_settings['PASSWORD'], db=self.db_settings['NAME'], loop=self.loop) self.cursor = await self.conn.cursor(aiomysql.cursors.DictCursor) return self async def __aexit__(self, exception, value, traceback): await self.cursor.close() self.conn.close() self.loop.close() async def query(self, sql, *args): await self.cursor.execute(sql, values) await self.conn.commit() rows = await self.cursor.fetchall() return list(rows) async def aiomysql_query(sql, *args): """ Mysql asynchronous connection wrapper """ loop = asyncio.get_event_loop() async with AsyncMysqlSession(loop) as mysql: db_result = await mysql.query(sql, *args) return db_result  References https://docs.python.org/3/library/asyncio-eventloop.html#asyncio.loop.shutdown_asyncgens https://docs.python.org/3/library/asyncio-eventloop.html#asyncio.get_event_loop
Question: friends: I am running a script in Linux: I can use the ps command get the process. ps -ef | grep "python test09.py&"  but, how can I know the pid of the running script by given key word python test09.py& using python code?  EDIT-01 I mean, I want to use the python script to find the running script python test09.py&'s pid.  EDIT-02 When I run anali's method I will get this error: Traceback (most recent call last): File "/Library/Frameworks/Python.framework/Versions/3.5/lib/python3.5/site-packages/psutil/_psosx.py", line 363, in catch_zombie yield File "/Library/Frameworks/Python.framework/Versions/3.5/lib/python3.5/site-packages/psutil/_psosx.py", line 429, in cmdline return cext.proc_cmdline(self.pid) ProcessLookupError: [Errno 3] No such process (originated from sysctl) During handling of the above exception, another exception occurred: Traceback (most recent call last): File "test11.py", line 29, in <module> print(get_pids_by_script_name('test09.py')) File "test11.py", line 15, in get_pids_by_script_name cmdline = proc.cmdline() File "/Library/Frameworks/Python.framework/Versions/3.5/lib/python3.5/site-packages/psutil/__init__.py", line 694, in cmdline return self._proc.cmdline() File "/Library/Frameworks/Python.framework/Versions/3.5/lib/python3.5/site-packages/psutil/_psosx.py", line 342, in wrapper return fun(self, *args, **kwargs) File "/Library/Frameworks/Python.framework/Versions/3.5/lib/python3.5/site-packages/psutil/_psosx.py", line 429, in cmdline return cext.proc_cmdline(self.pid) File "/Library/Frameworks/Python.framework/Versions/3.5/lib/python3.5/contextlib.py", line 77, in __exit__ self.gen.throw(type, value, traceback) File "/Library/Frameworks/Python.framework/Versions/3.5/lib/python3.5/site-packages/psutil/_psosx.py", line 376, in catch_zombie raise AccessDenied(proc.pid, proc._name) psutil.AccessDenied: psutil.AccessDenied (pid=1) 

Answer: If you just want the pid of the current script, then use os.getpid: import os pid = os.getpid()  However, below is an example of using psutil to find the pids of python processes running a named python script. This could include the current process, but the main use case is for examining other processes, because for the current process it is easier just to use os.getpid as shown above. sleep.py #!/usr/bin/env python import time time.sleep(100)  get_pids.py import os import psutil def get_pids_by_script_name(script_name): pids = [] for proc in psutil.process_iter(): try: cmdline = proc.cmdline() pid = proc.pid except psutil.NoSuchProcess: continue if (len(cmdline) >= 2 and 'python' in cmdline[0] and os.path.basename(cmdline[1]) == script_name): pids.append(pid) return pids print(get_pids_by_script_name('sleep.py'))  Running it: $ chmod +x sleep.py $ cp sleep.py other.py $ ./sleep.py & [3] 24936 $ ./sleep.py & [4] 24937 $ ./other.py & [5] 24938 $ python get_pids.py [24936, 24937]
Question: I have created a virtual environment with pyenv virtualenv 3.5.9 projectname for developing a django project. How can I set environment variables for my code to use? I tried to add the environment variable DATABASE_USER in /Users/developer/.pyenv/versions/projectname/bin/activate like this: export DATABASE_USER="dbuser"  When I tried to echo $DATABASE_USER an empty string gets printed. Tried to install zsh-autoenv And now I can echo $DATABASE_USER and get the value set in the .autoenv.zsh file. But I can't seem to get the environment variable to be available to my django code: If I try to os.getenv('DATABASE_USER', '') in the python shell inside the virtualenv, I get '' What could be wrong? Is the zsh-autoenv variables just available for the zsh shell and not python manage.py shell ?

Answer: I was wondering a similar thing, and I stumbled across a reddit thread where someone else had asked the same question, and eventually followed up noting some interesting finds. As you noticed, pyenv doesn't seem to actually use the bin/activate file. They didn't say what the activation method is, but like you, adding environment variables there yielded no results. In the end, they wound up installing autoenv, which bills itself as directory-based environments. It allows you to create an .env file in your directory, and when you cd to that directory, it runs the .env file. You can use it for environment variables, or you could add anything else to it. I noticed on the autoenv page that they say you should probably use direnv instead, as it has better features and is higher quality software. Neither of these are Python or pyenv specific, and if you call your python code from outside of the directory, they may not work. Since you're using pyenv, you're probably running your code from within the directory anyway, so I think there's a good chance either one could work.
Question: I want to know if it's possible to get an Amazon ECR container URI for a specific image programmatically (using AWS CLI or Python). For example, if I need the URL for the latest linear-learner (built-in model) image for the eu-central-1 region. Expected result: 664544806723.dkr.ecr.eu-central-1.amazonaws.com/linear-learner:latest  EDIT: I have found the solution with get_image_uri. It looks like this function will be depreceated and I don't know how to use ImageURIProvider instead.

Answer: The newer versions of SageMaker SDK have a more centralized API for getting the URIs: import sagemaker sagemaker.image_uris.retrieve("linear-learner", "eu-central-1")  which gives the expected result: 664544806723.dkr.ecr.eu-central-1.amazonaws.com/linear-learner:1
Question: I was wondering how is the "best" way to type-hint the axis-object of matplotlib-subplots. running from matplotlib import pyplot as plt f, ax = plt.subplots() print(type(ax))  returns <class 'matplotlib.axes._subplots.AxesSubplot'>  and running from matplotlib import axes print(type(axes._subplots)) print(type(axes._subplots.AxesSubplot))  yields <class 'module'> AttributeError: module 'matplotlib.axes._subplots' has no attribute 'AxesSubplots'  So far a solution for type-hinting that works is as follows: def multi_rocker( axy: type(plt.subplots()[1]), y_trues: np.ndarray, y_preds: np.ndarray, ): """ One-Vs-All ROC-curve: """ fpr = dict() tpr = dict() roc_auc = dict() n_classes = y_trues.shape[1] wanted = list(range(n_classes)) for i,x in enumerate(wanted): fpr[i], tpr[i], _ = roc_curve(y_trues[:, i], y_preds[:, i]) roc_auc[i] = round(auc(fpr[i], tpr[i]),2) extra = 0 for i in range(n_classes): axy.plot(fpr[i], tpr[i],) return  And the problem with it is that it isn't clear enough for code-sharing

Answer: As described in Type hints for context manager : import matplotlib.pyplot as plt def plot_func(ax: plt.Axes): ...
Question: I want to switch my notebook easily between different kernels. One use case is to quickly test a piece of code in tensorflow 2, 2.2, 2.3, and there are many similar use cases. However I prefer to define my environments as dockers these days, rather than as different (conda) environments. Now I know that you can start jupyter in a container, but that it not what I want. I would like to just click Kernel > use kernel > TF 2.2 (docker), and let jupyter connect to a kernel running in this container. Is something like that around? I have used livy to connect to remote spark kernels via ssh, so it feels like this should be possible.

Answer: Full disclosure: I'm the author of Dockernel. By using Dockernel Put the following in a file called Dockerfile, in a separate directory. FROM python:3.7-slim-buster RUN pip install --upgrade pip ipython ipykernel CMD python -m ipykernel_launcher -f $DOCKERNEL_CONNECTION_FILE  Then issue the following commands: docker build --tag my-docker-image /path/to/the/dockerfile/dir pip install dockernel dockernel install my-docker-image  You should now see "my-docker-image" option when creating a new notebook in Jupyter. Manually It is possible to do this kind of thing without much additional implementation/tooling, it just requires a bit of manual work:  Use the following Dockerfile:  FROM python:3.7-slim-buster RUN pip install --upgrade pip ipython ipykernel   Build the image using docker build --tag my-docker-image .  Create a directory for your kernelspec, e.g. ~/.local/share/jupyter/kernels/docker_test (%APPDATA%\jupyter\kernels\docker_test on Windows)  Put the following kernelspec into kernel.json file in the directory you created (Windows users might need to change argv a bit)   { "argv": [ "/usr/bin/docker", "run", "--network=host", "-v", "{connection_file}:/connection-spec", "my-docker-image", "python", "-m", "ipykernel_launcher", "-f", "/connection-spec" ], "display_name": "docker-test", "language": "python" }  Jupyter should now be able spin up a container using the docker image specified above.
Question: Python 3.9.0rc1 has been released today, according to the official website. Is there a way I can use it in an Anaconda environment? I tried conda create --name python39 python==3.9  But it says:  ERROR: Could not find a version that satisfies the requirement python==3.9 (from versions: none) ERROR: No matching distribution found for python==3.9  Edit: closing as duplicate rules out questions with no answer, and the self-accepted answer to the suggested duplicate does not answer the question. It says "use another distribution channel instead".

Answer: It's preferable to update Conda before installing Python 3.9: conda update -n base -c defaults conda  Then install a Python 3.9 environment. This works now: conda create --name python39 python==3.9
Question: After a lot of research, it seems like there is no good way to properly stop and resume training using a Tensorflow 2 / Keras model. This is true whether you are using model.fit() or using a custom training loop. There seem to be 2 supported ways to save a model while training:  Save just the weights of the model, using model.save_weights() or save_weights_only=True with tf.keras.callbacks.ModelCheckpoint. This seems to be preferred by most of the examples I've seen, however it has a number of major issues:  The optimizer state is not saved, meaning training resumption will not be correct. Learning rate schedule is reset - this can be catastrophic for some models. Tensorboard logs go back to step 0 - making logging essentually useless unless complex workarounds are implemented.   Save the entire model, optimizer, etc. using model.save() or save_weights_only=False. The optimizer state is saved (good) but the following issues remain:  Tensorboard logs still go back to step 0 Learning rate schedule is still reset (!!!) It is impossible to use custom metrics. This doesn't work at all when using a custom training loop - custom training loops use a non-compiled model, and saving/loading a non-compiled model doesn't seem to be supported.    The best workaround I've found is to use a custom training loop, manually saving the step. This fixes the tensorboard logging, and the learning rate schedule can be fixed by doing something like keras.backend.set_value(model.optimizer.iterations, step). However, since a full model save is off the table, the optimizer state is not preserved. I can see no way to save the state of the optimizer independently, at least without a lot of work. And messing with the LR schedule as I've done feels messy as well. Am I missing something? How are people out there saving/resuming using this API?

Answer: tf.keras.callbacks.experimental.BackupAndRestore API for resuming training from interruptions has been added for tensorflow>=2.3. It works great in my experience. Reference: https://www.tensorflow.org/api_docs/python/tf/keras/callbacks/experimental/BackupAndRestore
Question: It has been a long time when I asked this question and still didn't get an answers. I am trying to add infinite scroll down with Django but it is not working fine with the following code. I just paginating post by 10 and then its just showing me loading icon .it is not working when i am scrolling down. Can you guys figure it out what is wrong here ? views.py class PostListView(ListView): model = Post context_object_name = 'post_list' paginate_by = 10 def get_queryset(self): return Post.objects.filter(create_date__lte=timezone.now()).order_by('-create_date')  postlist.html {% extends 'base.html' %} {% block content %} <div class="container"> <div class="row infinite-container"> {% for post in post_list%} <div class="col-md-6 infinite-item"> <div class="card mb-4 shadow-sm"> <img class="img-thumbnail" src="{{post.image.url}}"/> <div class="card-body"> <h5>{{post.title}}</h5> <p class="card-text"> {{post.description|truncatewords:20}} </p> </div> </div> </div> {% endfor %} </div> {% if page_obj.has_next %} true #this is showing me true it also means that it has next page. <a class="infinite-more-link" href="?page={{page_obj.next_page_number}}"></a> {% endif %} <div class="d-flex justify-content-center" style="display:none;"> <div class="spinner-border" role="status"> <span class="sr-only">Loading...</span> </div> </div> </div> <script src="/static/js/jquery-2.2.4.min.js"></script> <script src="/static/js/jquery.waypoints.min.js"></script> <script src="/static/js/infinite.min.js"></script> <script> var infinite = new Waypoint.Infinite({ element: $('.infinite-container')[0], handler: function(direction) { }, offset: 'bottom-in-view', onBeforePageLoad: function () { $('.spinner-border').show(); }, onAfterPageLoad: function () { $('.spinner-border').hide(); } }); </script> {% endblock content %}  if more information is require than tell me in a comment session i will update my question with that information.

Answer: I was missing loading the static so load that by adding {% load static%}  below the content block <script src="{% static '/static/js/jquery-2.2.4.min.js'%}"></script> <script src="{% static '/static/js/jquery.waypoints.min.js'%}"></script> <script src="{% static '/static/js/infinite.min.js'%}"></script>
Question: I want to create a program that does something in which someone terminates the script by clicking the stop button in PyCharm. I tried from sys import exit def handler(signal_received, frame): # Handle any cleanup here print('SIGINT or CTRL-C detected. Exiting gracefully') exit(0) if __name__ == '__main__': signal(SIGINT, handler) print('Running. Press CTRL-C to exit.') while True: # Do nothing and hog CPU forever until SIGINT received. pass  from https://www.devdungeon.com/content/python-catch-sigint-ctrl-c. I tried on both Mac and Windows. On the Mac, PyCharm behaved as expected, when I click the stop button it catches the SIGINT. But on Windows, I did exactly the same thing, but it just straightly returns to me a Process finished with exit code -1. Is there something I can do to change to make the Windows behave like what on Mac? Any help is appreciated!

Answer: I don't think it's a strange question at all. On unix systems, pycham sends a SIGTERM, waits one second, then send a SIGKILL. On windows, it does something else to end the process, something that seems untrappable. Even during development you need a way to cleanly shut down a process that uses native resources. In my case, there is a CAN controller that, if not shut down properly, can't ever be opened again. My work around was to build a simple UI with a stop button that shuts the process down cleanly. The problem is, out of habit, from using pycharm, goland, and intellij, is to just hit the red, square button. Every time I do that I have to reboot the development system. So I think it is clearly also a development time question.
Question: I have a dataset with 84 Monthly Sales (from 01/2013 to 12/2019) - just months, not days. Month 01 | Sale 1 Month 02 | Sale 2 Month 03 | Sale 3 .... | ... Month 84 | Sale 84  By visualization it looks like that the model fits very well... but I need to check it.... So what I understood is that cross val does not support Months, and so what I did was convert to use it w/ days(although there is no day info into my original df)... I wanted to try my model w/ the first five years(60 months) and leave the 2 remaining years(24 months) to see how well the model is predicting.... So i did something like: cv_results = cross_validation( model = prophet, initial='1825 days', period='30 days', horizon = '60 days')  Does this make sense? I did not get the concept of cut off dates and forecast periods

Answer: I struggled with this for a while as well. But here is how it works. The initial model will be trained on the first 1,825 days of data. It will forecast the next 60 days of data (because horizon is set to 60). The model will then train on the initial period + the period (1,825 + 30 days in this case) and forecast the next 60 days. It will continued like this, adding another 30 days to the training data and then forecasting for the next 60 until there is no longer enough data to do this. In summary, period is how much data to add to the training data set in every iteration of cross-validation, and horizon is how far out it will forecast.
Question: During debugging or computationally heavy loops, i would like to see how my data processing evolves (for example in a line plot or an image). In matplotlib the code can redraw / update the figure with plt.cla() and then plt.draw() or plt.pause(0.001), so that i can follow the progress of my computation in real time or while debugging. How do I do that in plotly express (or plotly)?

Answer: So i think i essentially figured it out. The trick is to not use go.Figure() to create a figure, but go.FigureWidget() Which is optically the same thing, but behind the scenes it's not. documentation youtube video demonstration Those FigureWidgets are exactly there to be updated as new data comes in. They stay dynamic, and later calls can modify them. A FigureWidget can be made from a Figure: figure = go.Figure(data=data, layout=layout) f2 = go.FigureWidget(figure) f2 #display the figure  This is practical, because it makes it possible to use the simplified plotly express interface to create a Figure and then use this to construct a FigureWidget out of it. Unfortunately plotly express does not seem to have it's own simplified FigureWidget module. So one needs to use the more complicated go.FigureWidget.
Question: Trying to mock.patch a call to smtplib.SMTP.sendmail in a unittest. The sendmail method appears to be successfully mocked and we can query it as MagicMock, but the called and called_args attributes of the sendmail mock are not correctly updated. It seems likely I'm not applying the patch correctly. Here's a simplified example of what I'm trying: import unittest.mock with unittest.mock.patch('smtplib.SMTP', autospec=True) as mock: import smtplib smtp = smtplib.SMTP('localhost') smtp.sendmail('me', 'me', 'hello world\n') mock.assert_called() # <--- this succeeds mock.sendmail.assert_called() # <--- this fails  This example generates: AssertionError: Expected 'sendmail' to have been called.  If I alter the patch to smtp.SMTP.sendmail; eg: with unittest.mock.patch('smtplib.SMTP.sendmail.', autospec=True) as mock: ...  I can successfully access the called_args and called attributes of the mock in this case, but because the smtplib.SMTP initialization was allowed to take place, an actual smtp-session is established with a host. This is unittesting, and I'd prefer no actual networking take place.

Answer: I had the same issue today and forgot that I'm using a context, so just change mock.sendmail.assert_called()  to mock.return_value.__enter__.return_value.sendmail.assert_called()  That looks messy but here's my example: msg = EmailMessage() msg['From'] = 'no@no.com' msg['To'] = 'no@no.com' msg['Subject'] = 'subject' msg.set_content('content'); with patch('smtplib.SMTP', autospec=True) as mock_smtp: misc.send_email(msg) mock_smtp.assert_called() context = mock_smtp.return_value.__enter__.return_value context.ehlo.assert_called() context.starttls.assert_called() context.login.assert_called() context.send_message.assert_called_with(msg)