eshangj/stackoverflow_q_and_a_sample huggingface dataset:
Question: I use the following method a lot to append a single row to a dataframe. However, it has been deprecated. One thing I really like about it is that it allows you to append a simple dict object. For example: # Creating an empty dataframe df = pd.DataFrame(columns=['a', 'b']) # Appending a row df = df.append({ 'a': 1, 'b': 2 }, ignore_index=True)  Again, what I like most about this is that the code is very clean and requires very few lines. Now I suppose the recommended alternative is: # Create the new row as its own dataframe df_new_row = pd.DataFrame({ 'a': [1], 'b': [2] }) df = pd.concat([df, df_new_row])  So what was one line of code before is now two lines with a throwaway variable and extra cruft where I create the new dataframe. :( Is there a good way to do this that just uses a dict like I have in the past (that is not deprecated)?

Answer: Create a list with your dictionaries, if they are needed, and then create a new dataframe with df = pd.DataFrame.from_records(your_list). List's "append" method are very efficient and won't be ever deprecated. Dataframes on the other hand, frequently have to be recreated and all data copied over on appends, due to their design - that is why they deprecated the method
Question: I have problem with my pip . Lately I was getting error when I was trying to install any packages The Error was: ( Pyautogui ) Traceback (most recent call last): File "C:\Users\rati_\OneDrive\Desktop\PyAutoGUI-0.9.53.tar\PyAutoGUI-0.9.53\PyAutoGUI-0.9.53\setup.py", line 4, in <module> from setuptools import setup File "C:\Users\rati_\AppData\Local\Programs\Python\Python310\lib\site-packages\setuptools\__init__.py", line 12, in <module> from setuptools.extension import Extension File "C:\Users\rati_\AppData\Local\Programs\Python\Python310\lib\site-packages\setuptools\extension.py", line 7, in <module> from setuptools.dist import _get_unpatched File "C:\Users\rati_\AppData\Local\Programs\Python\Python310\lib\site-packages\setuptools\dist.py", line 16, in <module> import pkg_resources File "C:\Users\rati_\AppData\Local\Programs\Python\Python310\lib\site-packages\pkg_resources.py", line 29, in <module> import symbol ModuleNotFoundError: No module named 'symbol'  I reinstalled pip , python but couldn't fix the error ... There was no information online so I couldn't fix it. Any Tips?

Answer: Module symbol was a part of the standard library since the dawn of time. It was declared deprecated in Python 3.9 and finally removed in 3.10. For Python 3.10 one has to upgrade any 3rd-party library that imports symbol. In your case the libraries are pip/setuptools: pip install --upgrade pip setuptools  If upgrade is not possible or there is no newer version of libraries updated for Python 3.10 the only solution is to downgrade Python.
Question: I want to run selenium through chromium. I wrote this code: from selenium import webdriver from selenium.webdriver.chrome.options import Options options = Options() options.add_argument("start-maximized") options.add_argument("disable-infobars") options.add_argument("--disable-extensions") options.add_argument("--disable-gpu") options.add_argument("--disable-dev-shm-usage") options.add_argument("--no-sandbox") options.binary_location = "/snap/bin/chromium" driver = webdriver.Chrome(chrome_options=options)  But this code throws an error: selenium.common.exceptions.WebDriverException: Message: unknown error: DevToolsActivePort file doesn't exist Stacktrace: #0 0x55efd7355a23 <unknown> #1 0x55efd6e20e18 <unknown> #2 0x55efd6e46e12 <unknown>  The chromedriver of the correct version is in usr/bin. What am I doing wrong?

Answer: I solved the problem by reinstalling chromium through apt sudo apt install chromium-browser (before that it was installed through snap). My working code looks like this options = Options() options.add_argument("start-maximized") options.add_argument("disable-infobars") options.add_argument("--disable-extensions") options.add_argument("--disable-dev-shm-usage") options.add_argument("--no-sandbox") if headless: options.add_argument('--headless') options.binary_location = "/usr/bin/chromium-browser" self.driver = webdriver.Chrome(options=options)
Question: I have a Django model with the following structure: class BBPerformance(models.Model): marketcap_change = models.FloatField(verbose_name="marketcap change", null=True, blank=True) bb_change = models.FloatField(verbose_name="bestbuy change", null=True, blank=True) created_at = models.DateTimeField(verbose_name="created at", auto_now_add=True) updated_at = models.DateTimeField(verbose_name="updated at", auto_now=True)  I would like to have an Avg aggregate function on objects for every 3 days. for example I write a queryset that do this aggregation for each day or with something like TruncDay function. queryset = BBPerformance.objects.annotate(day=TruncDay('created_at')).values('day').annotate(marketcap_avg=Avg('marketcap_change'),bb_avg=Avg('bb_change')  How can I have a queryset of the aggregated value with 3-days interval and the index of the second day of that interval?

Answer: I guess it's impossible on DB level (and Trunc is DB level function) as only month, days weeks and so on are supported in Postgres and Oracle. So what I would suggest is to use TruncDay and then add python code to group those by 3 days.
Question: I am trying to run the example of a heatmap from: https://matplotlib.org/stable/gallery/images_contours_and_fields/image_annotated_heatmap.html When I am running the code in PyCharm (Professional) in an Anaconda3 environment it results in an error message.  TypeError: set_ticks() got an unexpected keyword argument 'labels' 

Answer: There is no need to update matplotlib version if you are using version of matplotlib 3.4.3. Here is the link for matplotlib 3.4.3 documentation and avoid using labels keyword in ax.set_xticks() function as said by JohnC
Question: I have a CSV file like this: Time Latitude Longitude 2021-09-12 23:13 44.63 -63.56 2021-09-14 23:13 43.78 -62 2021-09-16 23:14 44.83 -54.6  2021-09-12 23:13 is under Time column. I would like to open it using pandas. But there is a problem with the first column. It contains a space. If I open it using: import pandas as pd points = pd.read_csv("test.csv", delim_whitespace=True)  I get      Time Latitude Longitude     2021-09-12 23:13 44.630 -63.560   2021-09-14 23:13 43.780 -62.000   2021-09-16 23:14 44.830 -54.600     But I would like to skip the space in the first column in CSV (2021-09-12 23:13 should be under Time column) like:      Time Latitude Longitude     0 2021-09-12 23:13 44.630 -63.560   1 2021-09-14 23:13 43.780 -62.000   2 2021-09-16 23:14 44.830 -54.600     How can I ignore the first space when using pd.read_csv? Please do not stick to this csv file. This is a general question to skip (not to consider as a delimiter) the first space(s) in the first column. Because everyone knows that the first space is part of the time value, not a delimiter.

Answer: Ideally you should be parsing the first two parts as a datetime. By using a space as a delimiter, it would imply the header has three columns. The space after the date though is being seen as an extra column. A workaround is to skip the header entirely and supply your own column names. The parse_dates parameter can be used to tell Pandas to parse the first two columns as a single combined datetime object. For example: import pandas as pd points = pd.read_csv("test.csv", delimiter=" ", skipinitialspace=True, skiprows=1, index_col=None, parse_dates=[[0, 1]], names=["Date", "Time", "Latitude", "Longitude"]) print(points)  Should give you the following dataframe:  Date_Time Latitude Longitude 0 2021-09-12 23:13:00 44.63 -63.56 1 2021-09-14 23:13:00 43.78 -62.00 2 2021-09-16 23:14:00 44.83 -54.60
Question: I am trying to deploy my docker container on AWS Lambda. However, I use pdf2image package in my code which depends on poppler. To install poppler, I need to insert the following line in the Dockerfile. RUN apt-get install -y poppler-utils  This is the full view of the dockerfile. FROM ubuntu:18.04 RUN apt-get update RUN apt-get install -y poppler-utils RUN apt-get install python3 -y RUN apt-get install python3-pip -y RUN pip3 install --upgrade pip WORKDIR / COPY app.py . COPY requirements.txt . RUN pip3 install -r requirements.txt ENTRYPOINT [ "python3", "app.py" ]  However, to deploy on Lambda, I need to use AWS base python image for Lambda. This is my attempt to rewrite the above dockerfile to use the Lambda base image. FROM public.ecr.aws/lambda/python:3.6 # Cannot run the follow lines: apt-get: command not found # RUN apt-get update # RUN apt-get install -y poppler-utils COPY app.py . COPY requirements.txt . RUN pip install -r requirements.txt CMD ["app.handler"]  Based on the dockerfile above, you can see that the apt-get command cannot be run. Understandable because it is not from ubuntu image like I did earlier. My question is, how can I install the poppler in the Lambda base image?

Answer: It uses the yum package manager, so you can do the following instead: FROM public.ecr.aws/lambda/python:3.6 RUN yum install -y poppler-utils
Question: I'm trying to make a face detection model with CNN. I used codes that I made for number detection. When I use number images, program work. But, when I use my face images, I get an error that is: Unexpected result of train_function (Empty logs). Please use Model.compile(..., run_eagerly=True), or tf.config.run_functions_eagerly(True) for more information of where went wrong, or file a issue/bug to tf.keras. Notebook link: https://github.com/AkifCanSonmez/ImageProccessingCourse/blob/main/CNN/Number%20Classification%20Project/Building%20Model/Building%20Number%20Classification%20Model%20with%20Keras.ipynb Number image: Face image:

Answer: Your input images have a shape of (32,32,3) whil you first conv2D layer sets the inputshape to (32,32,1). Most likely your numbers have only 1 channel since they are grayscale, while you face images have 3 color channels. change: model.add(tf.keras.layers.Conv2D(input_shape = (32,32,1), filters = 8, kernel_size = (5,5),activation = "relu", padding = "same" ))  to model.add(tf.keras.layers.Conv2D(input_shape = (32,32,3), filters = 8, kernel_size = (5,5),activation = "relu", padding = "same" ))
Question: I am a beginner in the Django framework. I created my project created my app and test it, it work fine till I decided to add a template. I don't know where the error is coming from because I follow what Django docs say by creating folder name templates in your app folder creating a folder with your app name and lastly creating the HTML file in the folder. NOTE: other routes are working fine except the template Please view the screenshot of my file structure and error below.  ERROR  TemplateDoesNotExist at /blog/ Blog/index Request Method: GET Request URL: http://127.0.0.1:8000/blog/ Django Version: 4.0.1 Exception Type: TemplateDoesNotExist Exception Value: Blog/index Exception Location: C:\Python39\lib\site-packages\django\template\loader.py, line 19, in get_template Python Executable: C:\Python39\python.exe Python Version: 3.9.4 Python Path: ['C:\\Users\\Maxwell\\Desktop\\Django\\WebApp', 'C:\\Python39\\python39.zip', 'C:\\Python39\\DLLs', 'C:\\Python39\\lib', 'C:\\Python39', 'C:\\Users\\Maxwell\\AppData\\Roaming\\Python\\Python39\\site-packages', 'C:\\Python39\\lib\\site-packages', 'C:\\Python39\\lib\\site-packages\\win32', 'C:\\Python39\\lib\\site-packages\\win32\\lib', 'C:\\Python39\\lib\\site-packages\\Pythonwin'] Server time: Sun, 23 Jan 2022 12:04:18 +0000 Template-loader postmortem Django tried loading these templates, in this order: Using engine django: django.template.loaders.app_directories.Loader: C:\Users\Maxwell\Desktop\Django\WebApp\Blog\templates\ Blog\index (Source does not exist) django.template.loaders.app_directories.Loader: C:\Python39\lib\site-packages\django\contrib\admin\templates\ Blog\index (Source does not exist) django.template.loaders.app_directories.Loader: C:\Python39\lib\site-packages\django\contrib\auth\templates\ Blog\index (Source does not exist)  App/views.py from django.http import HttpResponse from django.shortcuts import render # Create your views here. def index(request,name): return HttpResponse(f'Hello {name}') def html(request): return render(request,' Blog/index.html') 

Answer: It seems that you render the template with Blog/index, but you need to specify the entire file name, so Blog/index.html and without a leading (or trailing) space: def html(request): return render(request, 'Blog/index.html')
Question: So I have a data science interview at Google, and I'm trying to prepare. One of the questions I see a lot (on Glassdoor) from people who have interviewed there before has been: "Write code to generate random normal distribution." While this is easy to do using numpy, I know sometimes Google asks the candidate to code without using any packages or libraries, so basically from scratch. Any ideas?

Answer: According to the Central Limit Theorem a normalised summation of independent random variables will approach a normal distribution. The simplest demonstration of this is adding two dice together. So maybe something like: import random import matplotlib.pyplot as plt def pseudo_norm(): """Generate a value between 1-100 in a normal distribution""" count = 10 values = sum([random.randint(1, 100) for x in range(count)]) return round(values/count) dist = [pseudo_norm() for x in range(10_000)] n_bins = 100 fig, ax = plt.subplots() ax.set_title('Pseudo-normal') hist = ax.hist(dist, bins=n_bins) plt.show()  Which generates something like:
Question: In Julia, it is possible to export a function (or a variable, struct, etc.) of a module, after which it can be called in another script without the namespace (once it has been imported). For example: # helpers.jl module helpers function ordinary_function() # bla bla bla end function exported_function() # bla bla bla end export exported_function() end  # main.jl include("helpers.jl") using .helpers #we have to call the non-exported function with namespace helpers.ordinary_function() #but we can call the exported function without namespace exported_function()  Is this feature available in Python?

Answer: In Python importing is easier than this: # module.py def foo(): pass # file.py from module import foo foo() # file2.py from file import foo foo()  This works with classes too.  In Python you can also do something like this: import module # You have to call like this: module.foo()   When you import a module, all the functions the module imported are considered part of the module. Using the example below: # file.py import module # file2.py import file file.module.foo() # or... from file import module module.foo() # ...or... from file.module import foo foo()   Notice that in Python the export is not needed. Look at the documentation: no export keyword exists in Python.
Question: I have a telegram bot and I want to send a message  in which the error message will be returned to me my code is :  path = 'C:\\Bot\\Log\\aaa\\*.log' files = glob.glob(path) nlines= 0 data = "Servers : \n" for name in files: with open(name) as f: for line in f : nlines += 1 if (line.find("Total") >= 0): data += line for i in range(5): data += next(f) data += f'\n{emoji.emojize(":blue_heart:")} ----------------------------------------------------{emoji.emojize(":blue_heart:")}\n' if (line.find("Source") >= 0): data += line query.edit_message_text( text=f"{data}", reply_markup=build_keyboard(number_list), )  my error is : telegram.error.BadRequest: Message_too_long  According to this code model, how can I send my message to the bot?

Answer: its still an open issue, but you can split your request for 4089 chars per send you have 2 options: if len(info) > 4096: for x in range(0, len(info), 4096): bot.send_message(message.chat.id, info[x:x+4096]) else: bot.send_message(message.chat.id, info)  or msgs = [message[i:i + 4096] for i in range(0, len(message), 4096)] for text in msgs: update.message.reply_text(text=text)
Question: How do safely store passwords and API keys within an .env file and properly parse them? using python? I want to store passwords that I do not wish to push into public repos.

Answer: You may parse the key values of an .env file you can use os.getenv(key) where you replace key with the key of the value you want to access Suppose the contents of the .env files are : A=B FOO=BAR SECRET=VERYMUCH  You can parse the content like this : import os print(os.getenv("A")) print(os.getenv("FOO")) print(os.getenv("SECRET")) # And so on...  Which would result in (output / stdout) : B BAR VERYMUCH  Now, if you are using git version control and you never want yourself to push an environment file accidentally then add this to your .gitignore file and they'll be conveniently ignored. .env  (Although you should make your life a bit easier by using an existing .gitignore template all of which by default ignore .env files) Lastly :  you can load .env just like a text file by doing :  with open(".env") as env: ...  and then proceed to manually parse it using regex  In case for some reason if .env files aren't detected by your python script then use this module python-dotenv  You may make use of the aforementioned library like this : from dotenv import load_dotenv load_dotenv() # take environment variables from .env. # Code of your application, which uses environment variables (e.g. from `os.environ` or # `os.getenv`) as if they came from the actual environment.
Question: I have this code: from typing import Callable, Any class Test(classmethod): def __init__(self, f: Callable[..., Any]): super().__init__(f) def __get__(self,*args,**kwargs): print(args) # why out put is (None, <class '__main__.A'>) where form none why no parameter 123 # where was it called return super().__get__(*args,**kwargs) class A: @Test def b(cls,v_b): print(cls,v_b) A.b(123)  Why the output is (None, <class '__main__.A'>)? Where did None come form and why is it not the parameter 123, which is the value I called it with?

Answer: The __get__ method is called when the method b is retrieved from the A class. It has nothing to do with the actual calling of b. To illustrate this, separate the access to b from the actual call of b: print("Getting a reference to method A.b") method = A.b print("I have a reference to the method now. Let's call it.") method()  This results in this output: Getting a reference to method A.b (None, <class '__main__.A'>) I have a reference to the method now. Let's call it. <class '__main__.A'> 123  So you see, it is normal that the output in __get__ does not show anything about the argument you call b with, because you haven't made the call yet.  The output None, <class '__main__.A'> is in line with the Python documentation on __get__:  object.__get__(self, instance, owner=None) Called to get the attribute of the owner class (class attribute access) or of an instance of that class (instance attribute access). The optional owner argument is the owner class, while instance is the instance that the attribute was accessed through, or None when the attribute is accessed through the owner.  In your case you are using it for accessing an attribute (b) of a class (A) -- not of an instance of A -- so that explains the instance argument is None and the owner argument is your class A.  The second output, made with print(cls,v_b), will print <class '__main__.A'> for cls, because that is what happens when you call class methods (as opposed to instance methods). Again, from the documentation:  When a class attribute reference (for class C, say) would yield a class method object, it is transformed into an instance method object whose __self__ attribute is C.  Your case is described here, where A is the class, and so the first parameter (which you called cls) will get as value A.
Question: I am using df.append() in my code to append the percentage change across dataframe columns. With df.append() being depreciated in pandas 1.4, I am trying to use pd.concat but I am not able to replicate the output. So here is what I have now: import numpy as np import pandas as pd df = pd.DataFrame({"A": ["foo", "foo", "foo", "foo", "foo", "bar", "bar", "bar", "bar"], "B": ["one", "one", "one", "two", "two", "one", "one", "two", "two"], "C": ["small", "large", "large", "small", "small", "large", "small", "small", "large"], "D": [1, 2, 2, 3, 3, 4, 5, 6, 7], "E": [2, 4, 5, 5, 6, 6, 8, 9, 9]}) table = pd.pivot_table(df, values='D', index=['A', 'B'], columns=['C'], aggfunc=np.sum, fill_value=0, margins=True) table.append( (table .iloc[-1] .pct_change(periods=1, fill_method=None) .fillna('') .apply(lambda x: '{:.1%}'.format(x) if x else '') ) )  The output is:  C large small All A B bar one 4 5 9 two 7 6 13 foo one 4 1 5 two 0 6 6 All 15 18 33 20.0% 83.3%  which is what I am after, but I am getting the depreciated warning, FutureWarning: The frame.append method is deprecated and will be removed from pandas in a future version. Use pandas.concat instead.table.append()  I changed my code to use pd.concat(), as follow: pd.concat([table, (table .iloc[-1] .pct_change(periods=1, fill_method=None) .fillna('') .apply(lambda x: '{:.1%}'.format(x) if x else '') )], )  now I am getting:  large small All 0 (bar, one) 4.0 5.0 9.0 NaN (bar, two) 7.0 6.0 13.0 NaN (foo, one) 4.0 1.0 5.0 NaN (foo, two) 0.0 6.0 6.0 NaN (All, ) 15.0 18.0 33.0 NaN large NaN NaN NaN small NaN NaN NaN 20.0% All NaN NaN NaN 83.3%  which is not what I expected - note the percentage changes (20% and 83.3%) compared to the output from append. Any input would be appreciated.

Answer: Use pd.concat like this. Convert your inner command to df using Series.to_frame and then transpose it using df.T: In [74]: pd.concat([table, table.iloc[-1].pct_change(periods=1, fill_method=None).fillna('').apply(lambda x: '{:.1%}'.format(x) if x else '').to_frame().T]) Out[74]: C large small All A B bar one 4 5 9 two 7 6 13 foo one 4 1 5 two 0 6 6 All 15 18 33 20.0% 83.3%
Question: I am using Poetry in some of my python projects. It's not unusual that at some stage I want to rename the root folder of my project. When I do that and run poetry shell poetry creates a new virtual environment. But I don't want a new virtual environment, I just want to keep using the existing virtual environment. I know I can manually activate the old one by running source {path to the old venv}/bin/activate but then I would have to keep track of the old environment name separately and refrain from using poetry shell. Is there something I can do about this? It's quite time consuming to start installing the dependencies again, pointing an IDE to the new environment and deleting the old virtual env, just because you have changed the root folder name - something that can happen multiple times. This question suggests that there is no solution to the problem but would want to confirm this because to me this seems quite annoying issue.

Answer: One option is to enable the virtualenvs.in-project option, e.g. by running poetry config virtualenvs.in-project true   If set to true, the virtualenv wil [sic] be created and expected in a folder named .venv within the root directory of the project.  This will cause Poetry to create new environments in $project_root/.venv/. If you rename the project directory the environment should continue to work.