eshangj/stackoverflow_q_and_a_sample huggingface dataset:
Question: Hello fellow AWS contributors, I’m currently working on a project to set up an example of connecting a Lambda function to our PostgreSQL database hosted on RDS. I tested my Python + SQL code locally (in VS code and DBeaver) and it works perfectly fine with including only basic credentials(host, dbname, username password). However, when I paste the code in Lambda function, it gave me all sorts of errors. I followed this template and modified my code to retrieve the credentials from secret manager instead. I’m currently using boto3, psycopg2, and secret manager to get credentials and connect to the database. List of errors I’m getting-  server closed the connection unexpectedly. This probably means the server terminated abnormally before or while processing the request could not connect to server: Connection timed out. Is the server running on host “db endpoint” and accepting TCP/IP connections on port 5432? FATAL: no pg_hba.conf entry for host “ip:xxx”, user "userXXX", database "dbXXX", SSL off  Things I tried -  RDS and Lambda are in the same VPC, same subnet, same security group. IP address is included in the inbound rule Lambda function is set to run up to 15 min, and it always stops before it even hits 15 min I tried both database endpoint and database proxy endpoint, none of it works.  It doesn’t really make sense to me that when I run the code locally, I only need to provide the host, dbname, username, and password, that’s it, and I’m able to write all the queries and function I want. But when I throw the code in lambda function, it’s requiring all these secret manager, VPC security group, SSL, proxy, TCP/IP rules etc. Can someone explain why there is a requirement difference between running it locally and on lambda? Finally, does anyone know what could be wrong in my setup? I'm happy to provide any information in related to this, any general direction to look into would be really helpful. Thanks!

Answer: Following the directions at the link below to build a specific psycopg2 package and also verifying the VPC subnets and security groups were configured correctly solved this issue for me. I built a package for PostgreSQL 10.20 using psycopg2 v2.9.3 for Python 3.7.10 running on an Amazon Linux 2 AMI instance. The only change to the directions I had to make was to put the psycopg2 directory inside a python directory (i.e. "python/psycopg2/") before zipping it -- the import psycopg2 statement in the Lambda function failed until I did that.  https://kalyanv.com/2019/06/10/using-postgresql-with-python-on-aws-lambda.html  This the VPC scenario I'm using. The Lambda function is executing inside the Public Subnet and associated Security Group. Inbound rules for the Private Subnet Security Group only allow TCP connections to 5432 for the Public Subnet Security Group.  https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/USER_VPC.Scenarios.html#USER_VPC.Scenario1
Question: I have defined an empty data frame with: insert_row = { "Date": dtStr, "Index": IndexVal, "Change": IndexChnge, } data = { "Date": [], "Index": [], "Change": [], } df = pd.DataFrame(data) df = df.append(insert_row, ignore_index=True) df.to_csv(r"C:\Result.csv", index=False) driver.close()  But I get the below deprecation warning not to use df.append every time I run the code  Can anyone suggest how to get rid of this warning by using pandas.concat?

Answer: Create a dataframe then concat: insert_row = { "Date": '2022-03-20', "Index": 1, "Change": -2, } df = pd.concat([df, pd.DataFrame([insert_row])]) print(df) # Output Date Index Change 0 2022-03-20 1.0 -2.0
Question: { "type": "service_account", "project_id": "project_id", "private_key_id": "private_key_id", "private_key": "-----BEGIN PRIVATE KEY-----\n", "client_email": "email", "client_id": "id", "auth_uri": "uri_auth", "token_uri": "token_urin", "auth_provider_x509_cert_url": "auth_provider_x509_cert_url", "client_x509_cert_url": "client_x509_cert_url" }  I tried encoding and decoding the JSON but it didn't work I even tried using /// in place of " " So I am using sheets-api. What I want to achieve is loading the path-for-json-file from .env variable scope=['https://spreadsheets.google.com/feeds', 'https://www.googleapis.com/auth/drive', 'https://www.googleapis.com/auth/drive.file', 'https://www.googleapis.com/auth/spreadsheets' ] credentials = ServiceAccountCredentials.from_json_keyfile_name(r"path-for-json-file", scope) client = gspread.authorize(credentials) 

Answer: Assuming your JSON file is creds.json creds.json { "type": "service_account", "project_id": "project_id", "private_key_id": "private_key_id", "private_key": "-----BEGIN PRIVATE KEY-----\n", "client_email": "email", "client_id": "id", "auth_uri": "uri_auth", "token_uri": "token_urin", "auth_provider_x509_cert_url": "auth_provider_x509_cert_url", "client_x509_cert_url": "client_x509_cert_url" }  main.py import json data = json.load(open('creds.json')) f = open(".env", "x") for key, value in data.items(): f.write(f"{key.upper()}={value}\n")  creds.env will be generated TYPE=service_account PROJECT_ID=project_id PRIVATE_KEY_ID=private_key_id PRIVATE_KEY=-----BEGIN PRIVATE KEY----- CLIENT_EMAIL=email CLIENT_ID=id AUTH_URI=uri_auth TOKEN_URI=token_urin AUTH_PROVIDER_X509_CERT_URL=auth_provider_x509_cert_url CLIENT_X509_CERT_URL=client_x509_cert_url  create_keyfile_dict() basically returns a dict called variable_keys from dotenv import load_dotenv load_dotenv() def create_keyfile_dict(): variables_keys = { "type": os.getenv("TYPE"), "project_id": os.getenv("PROJECT_ID"), "private_key_id": os.getenv("PRIVATE_KEY_ID"), "private_key": os.getenv("PRIVATE_KEY"), "client_email": os.getenv("CLIENT_EMAIL"), "client_id": os.getenv("CLIENT_ID"), "auth_uri": os.getenv("AUTH_URI"), "token_uri": os.getenv("TOKEN_URI"), "auth_provider_x509_cert_url": os.getenv("AUTH_PROVIDER_X509_CERT_URL"), "client_x509_cert_url": os.getenv("CLIENT_X509_CERT_URL") } return variables_keys scope=['https://spreadsheets.google.com/feeds', 'https://www.googleapis.com/auth/drive', 'https://www.googleapis.com/auth/drive.file', 'https://www.googleapis.com/auth/spreadsheets' ] credentials = ServiceAccountCredentials.from_json_keyfile_name(create_keyfile_dict(), scope) client = gspread.authorize(credentials)
Question: This is how it is structured  The code inside apps.py of accounts folder file is from django.apps import AppConfig class AccountsConfig(AppConfig): default_auto_field = 'django.db.models.BigAutoField' name = "apps.accounts"  The code inside Settings is INSTALLED_APPS = [ 'django.contrib.admin', 'django.contrib.auth', 'django.contrib.contenttypes', 'django.contrib.sessions', 'django.contrib.messages', 'django.contrib.staticfiles', 'mysite.apps.accounts', ]  I tried changing 'mysite.apps.accounts', to 'mysite.apps.AccountsConfig', and changing name = "apps.accounts" to name = "accounts" I am new to Django and was following How to make a website with Python and Django - MODELS AND MIGRATIONS (E04) tutorial. Around 16:17 is where my error comes up when I enter python manage.py makemigrate to the vscode terminal The error is  ImproperlyConfigured( django.core.exceptions.ImproperlyConfigured: Cannot import 'apps.accounts'. Check that 'mysite.apps.accounts.apps.AccountsConfig.name' is correct. Someone please help me. 

Answer: The solution was quite counterintuitive. You have to delete the class AccountsConfig(AppConfig): default_auto_field = 'django.db.models.BigAutoField' name = "accounts"  from apps.py\accounts\apps\mysite. Then run python manage.py makemigrations and 2 new models 'UserPersona' and 'UserProfile' are created. the output in the terminal: mysite\apps\accounts\migrations\0001_initial.py - Create model UserPersona - Create model UserProfile
Question: i have this database called db in pandas  index win loss moneywin moneyloss player1 5 1 300 100 player2 10 5 650 150 player3 17 6 1100 1050 player11 1010 105 10650 10150 player23 1017 106 101100 101050  and i want to add the elements of list1 to the elements of db list1 = [[player1,105,101,10300,10100],[player3,17,6,1100,1050]]  so the results would be db2 index win loss moneywin moneyloss player1 110 102 10600 10200 player2 10 5 650 150 player3 34 12 2200 2100 player11 1010 105 10650 10150 player23 1017 106 101100 101050  how can i go about it?

Answer: Solution 1: Create a dataframe from list1 then concat it with the given dataframe then group by index and aggregate the remaining columns using sum df1 = pd.DataFrame(list1, columns=df.columns) df_out = pd.concat([df, df1]).groupby('index', sort=False).sum()  Solution 2: Create a dataframe from list1 then add it with the given dataframe using common index df1 = pd.DataFrame(list1, columns=df.columns) df_out = df.set_index('index').add(df1.set_index('index'), fill_value=0)  Result: print(df_out) win loss moneywin moneyloss index player1 110 102 10600 10200 player2 10 5 650 150 player3 34 12 2200 2100 player11 1010 105 10650 10150 player23 1017 106 101100 101050
Question: I am defining a function that gets pdf in bytes, so I wrote: def documents_extractos(pdf_bytes: bytes): pass  When I call the function and unfortunately pass a wrong type, instead of bytes let's say an int, why I don't get an error? I have read the documentation regarding typing but I don't get it. Why is the purpose of telling the function that the variable shoudl be bytes but when you pass and int there is no error? This could be handle by a isinstance(var, <class_type>) right? I don't understand it =(

Answer: Type hints are ignored at runtime. At the top of the page, the documentation that you've linked contains a note that states (emphasis mine):  The Python runtime does not enforce function and variable type annotations. They can be used by third party tools such as type checkers, IDEs, linters, etc.  The purpose of type hints is for static typechecking tools (e.g. mypy), which use static analysis to verify that your code respects the written type hints. These tools must be run as a separate process. Their primary use is to ensure that new changes in large codebases do not introduce potential typing issues (which can eventually become latent bugs that are difficult to resolve). If you want explicit runtime type checks (e.g. to raise an Exception if a value of a wrong type is passed into a function), use isinstance().
Question: I want to estimate the fourier transform for a given image of size BxCxWxH In previous torch version the following did the job: fft_im = torch.rfft(img, signal_ndim=2, onesided=False)  and the output was of size: BxCxWxHx2  However, with the new version of rfft : fft_im = torch.fft.rfft2(img, dim=2, norm=None)  I do not get the same results. Do I miss something?

Answer: A few issues  The dim argument you provided is an invalid type, it should be a tuple of two numbers or should be omitted. Really PyTorch should raise an exception. I would argue that the fact this ran without exception is a bug in PyTorch (I opened a ticket stating as much). PyTorch now supports complex tensor types, so FFT functions return those instead of adding a new dimension for the real/imaginary parts. You can use torch.view_as_real to convert to the old representation. Also worth pointing out that view_as_real doesn't copy data since it returns a view so shouldn't slow things down in any noticeable way. PyTorch no longer gives the option of disabling one-sided calculation in RFFT. Probably because disabling one-sided makes the result identical to torch.fft.fft2, which is in conflict with the 13th aphorism of PEP 20. The whole point of providing a special real-valued version of the FFT is that you need only compute half the values for each dimension, since the rest can be inferred via the Hermition symmetric property.  So from all that you should be able to use fft_im = torch.view_as_real(torch.fft.fft2(img))  Important If you're going to pass fft_im to other functions in torch.fft (like fft.ifft or fft.fftshift) then you'll need to convert back to the complex representation using torch.view_as_complex so those functions don't interpret the last dimension as a signal dimension.
Question: I use textmate for make pdf file in latex. After the update of macOS Monterey version 12.3, the minimal version of python (/usr/bin/python) has disappeared : the compilation don't work now. I try to change in the textmate's files /usr/bin/python by /usr/bin/python3 (I have only this python folder) but that always don't work. the error say me ti change the compilation command which is this :  #!/usr/bin/env ruby18 # coding: utf-8 require ENV["TM_SUPPORT_PATH"] + "/lib/tm/process" require ENV["TM_SUPPORT_PATH"] + "/lib/tm/htmloutput" require ENV["TM_SUPPORT_PATH"] + "/lib/tm/save_current_document" # To enable the typesetting of unsaved documents, you must change the “Save” setting of # this command to “Current File” and add the variable TM_LATEX_AUTOSAVE to TextMate's # Shell Variables preferences. Be warned that your document must be encoded as UTF-8 if # you exercise this option — becauseTextMate.save_current_document cannot know the file # encoding you prefer. TextMate.save_current_document unless ENV["TM_LATEX_AUTOSAVE"].nil? texmate = ENV["TM_BUNDLE_SUPPORT"] + "/bin/texmate.py" engine_version = TextMate::Process.run(texmate, "version") TextMate::HTMLOutput.show(:title => "Typesetting “#{ENV["TM_DISPLAYNAME"] || File.basename(ENV["TM_FILEPATH"])}”…", :sub_title => engine_version) do |io| TextMate::Process.run(texmate, 'latex', :interactive_input => false) do |line| io << line end end ::Process.exit($?.exitstatus || 0) # exitstatus is nil if our process is prematurely terminated (SIGINT)  Thank you very much for your help. PS : The compilation work with texshop, I don't think it is a latex problem

Answer: The LaTeX-Bundle of TextMate was not updated in time for the release of MacOS 12.3. You can fix it as follows:  Download and install Python 3 (https://www.python.org/downloads/) /usr/bin/python3 -m pip install pyobjc --user cd ~/Library/Application\ Support/TextMate/Managed/Bundles/LaTeX.tmbundle/Support/bin Change “python” to “python3” in the header of all .py files (configure.py, btexdoc.py, texmate.py, texparser.py)
Question: I have multiple python versions managed by pyenv. I want to upgrade one of my virtual environments from 3.7.13 to 3.10.3 with the ‘—upgrade’ option as: >deactivate >pyenv local 3.10.3 >python3 -m venv --upgrade .venv >. .venv/bin/activate > python -V Python 3.7.13  I expect the '—upgrade' would change the python version to 3.10.3 but it did not it stayed with 3.7.13 I understand it may be easier just discard and recreate the virtual environment, but I really want to learn how '—upgrade' should work

Answer: If you read the official documentation of the venv module, then the description of the --upgrade option is very specific: "... assuming Python has been upgraded in-place." I think this implies that it has to be the same Python installation that you originally created the virtual environment with, for the --upgrade flag to work. Each version of Python installed by pyenv is installed separately, so I wouldn't expect the --upgrade flag to work in this case. That being said, as far as I know, venv does little more than installing a couple of basic scripts and configuration files, and some bunch of symbolic links. The source code of the venv module seems fairly straightforward, and all that the --upgrade switch does is skip the setup scripts. I think you could manually "hack" your way through this by changing some symbolic links and changing some directory names here and there. However, it's not how venv should be used. So, yeah, save yourself the misery, and discard the old virtual environment and just build a new one.
Question: I am trying to access a site using selenium Python. But the site is checking and checking continuously by cloudflare. No other page is coming. Check the screenshot here.  I have tried undetected chrome but it is not working at all.

Answer: By undetected chrome do you mean undetected chromedriver?: Anyways, undetected-chromedriver works for me: Undetected chromedriver Github: https://github.com/ultrafunkamsterdam/undetected-chromedriver pip install undetected-chromedriver  Code that gets a cloudflare protected site: import undetected_chromedriver as uc driver = uc.Chrome(use_subprocess=True) driver.get('https://nowsecure.nl')  My POV    Quick setup code that logs into your google account: Github: https://github.com/xtekky/google-login-bypass import undetected_chromedriver as uc from selenium.webdriver.common.by import By from selenium.webdriver.support.ui import WebDriverWait from selenium.webdriver.support import expected_conditions as EC # ---------- EDIT ---------- email = 'email\n' # replace email password = 'password\n' # replace password # ---------- EDIT ---------- driver = uc.Chrome(use_subprocess=True) wait = WebDriverWait(driver, 20) url = 'https://accounts.google.com/ServiceLogin?service=accountsettings&continue=https://myaccount.google.com%3Futm_source%3Daccount-marketing-page%26utm_medium%3Dgo-to-account-button' driver.get(url) wait.until(EC.visibility_of_element_located((By.NAME, 'identifier'))).send_keys(email) wait.until(EC.visibility_of_element_located((By.NAME, 'password'))).send_keys(password) print("You're in!! enjoy") # [ ---------- paste your code here ---------- ]
Question: Can you help me replace append with concat in this code? saida = pd.DataFrame() for x, y in lCodigos.items(): try: df = consulta_bc(x) logging.info(f'Indice {y} lido com sucesso.') except Exception as err: logging.error(err) logging.warning('Rotina Indice falhou!') exit() df['nome'] = y saida = saida.append(df) print(saida) 

Answer: Just save the "dataframe parts" using a list and use pd.concat on that list of dataframes at the end: saida = list() # Now use a list for x, y in lCodigos.items(): # ... your original code saida.append(df) saida = pd.concat(saida) # You can now create the dataframe
Question: import requests response = requests.get("https://gitlab.com/api/v4/users/ahmed_sh/projects") print(response.status_code) # 200 print(response.text) # [] print(response.json()) # []  I'm trying to get a list of my GitLab repo projects using python API, but the outputs are nothing! Although, when I use the browser, I got a non-empty response. How can I solve this problem?

Answer: This is because you don't have any public projects in your user namespace. If you want to see your private projects in your namespace, you'll need to authenticate with the API by passing a personal access token in the PRIVATE-TOKEN header. Note, this also won't show projects you work on in other namespaces. headers = {'PRIVATE-TOKEN': 'Your API key here!'} resp = requests.get('https://gitlab.com/api/v4/users/ahmed_sh/projects', headers=headers) print(resp.json())
Question: I would normally install a Python repository from Github using (for example): pip install git+git://github.com/Artory/drf-hal-json@master  And concordantly, my "requirements.txt" would have git+git://github.com/Artory/drf-hal-json@master in it somewhere. This failed today. The full traceback is below, but the relevant part is: The unauthenticated git protocol on port 9418 is no longer supported.  Thanks Microsoft. The traceback points to this link about the update. Most of the page at the link talks about how the update is unlikely to affect many people (thanks again Microsoft), and the rest of it involves cryptography that I'm far too noob to understand. The section titled "git://" simply reads:  On the Git protocol side, unencrypted git:// offers no integrity or authentication, making it subject to tampering. We expect very few people are still using this protocol, especially given that you can’t push (it’s read-only on GitHub). We’ll be disabling support for this protocol.  This doesn't help me understand how to update my requirements.txt to make it work again. Can you tell me how to update my requirements.txt to make it work again? Full traceback below: (venv) neil~/Documents/Code/web_app$ pip install git+git://github.com/Artory/drf-hal-json@master Collecting git+git://github.com/Artory/drf-hal-json@master Cloning git://github.com/Artory/drf-hal-json (to revision master) to /tmp/pip-req-build-zowfe130 Running command git clone -q git://github.com/Artory/drf-hal-json /tmp/pip-req-build-zowfe130 fatal: remote error: The unauthenticated git protocol on port 9418 is no longer supported. Please see https://github.blog/2021-09-01-improving-git-protocol-security-github/ for more information. WARNING: Discarding git+git://github.com/Artory/drf-hal-json@master. Command errored out with exit status 128: git clone -q git://github.com/Artory/drf-hal-json /tmp/pip-req-build-zowfe130 Check the logs for full command output. ERROR: Command errored out with exit status 128: git clone -q git://github.com/Artory/drf-hal-json /tmp/pip-req-build-zowfe130 Check the logs for full command output. WARNING: You are using pip version 21.2.4; however, version 22.0.4 is available. You should consider upgrading via the '/home/neil/Documents/Code/web_app/venv/bin/python -m pip install --upgrade pip' command. 

Answer: In the URL you give to pip, the git+git says to access a Git repository (the first git) over the unauthenticated git protocol (the second git). Assuming you want to continue to use anonymous access here, you can simply rewrite the command to use git+https instead, which access a Git repository over the secure HTTPS protocol. So your command would look like this: $ pip install git+https://github.com/Artory/drf-hal-json@master  I just tested in a VM, and that appears to work. If you have other such URLs, changing the same way should be effective.
Question: How do you type hint scipy sparse matrices, such as CSR, CSC, LIL etc.? Below is what I have been doing, but it doesn't feel right: def foo(mat: scipy.sparse.csr.csr_matrix): # Do whatever  What do we do if our function can accept multiple types of scipy sparse matrices (i.e any of them)?

Answer: All of csr, csc, lil are types of scipy.sparse.base.spmatrix: from scipy import sparse c1 = sparse.lil.lil_matrix c2 = sparse.csr.csr_matrix c3 = sparse.csc.csc_matrix print(c1.__bases__[0]) print(c2.__base__.__base__.__base__) print(c3.__base__.__base__.__base__)  Output: <class 'scipy.sparse.base.spmatrix'> <class 'scipy.sparse.base.spmatrix'> <class 'scipy.sparse.base.spmatrix'>  So you have an option to: def foo(mat: scipy.sparse.base.spmatrix): # Do whatever
Question: I know what namespaces are. But when running import argparse parser = argparse.ArgumentParser() parser.add_argument('bar') parser.parse_args(['XXX']) # outputs: Namespace(bar='XXX')  What kind of object is Namespace(bar='XXX')? I find this totally confusing. Reading the argparse docs, it says "Most ArgumentParser actions add some value as an attribute of the object returned by parse_args()". Shouldn't this object then appear when running globals()? Or how can I introspect it?

Answer: Samwise's answer is very good, but let me answer the other part of the question.  Or how can I introspect it?  Being able to introspect objects is a valuable skill in any language, so let's approach this as though Namespace is a completely unknown type. >>> obj = parser.parse_args(['XXX']) # outputs: Namespace(bar='XXX')  Your first instinct is good. See if there's a Namespace in the global scope, which there isn't. >>> Namespace Traceback (most recent call last): File "<stdin>", line 1, in <module> NameError: name 'Namespace' is not defined  So let's see the actual type of the thing. The Namespace(bar='XXX') printer syntax is coming from a __str__ or __repr__ method somewhere, so let's see what the type actually is. >>> type(obj) <class 'argparse.Namespace'>  and its module >>> type(obj).__module__ 'argparse'  Now it's a pretty safe bet that we can do from argparse import Namespace and get the type. Beyond that, we can do >>> help(argparse.Namespace)  in the interactive interpreter to get detailed documentation on the Namespace class, all with no Internet connection necessary.