eshangj/stackoverflow_q_and_a_sample huggingface dataset:
Question: I'm having a hard time understanding from the documentation exactly what typing.Annotated is good for and an even harder time finding explanations/examples outside the documentation. Or does it "being good for something" depend entirely on what third party libraries you're using? In what (real-world) context would you use Annotated?

Answer: Annotated in python allows developers to declare the type of a reference and provide additional information related to it. name: Annotated[str, "first letter is capital"]  This tells that name is of type str and that name[0] is a capital letter. On its own Annotated does not do anything other than assigning extra information (metadata) to a reference. It is up to another code, which can be a library, framework or your own code, to interpret the metadata and make use of it. For example, FastAPI uses Annotated for data validation: def read_items(q: Annotated[str, Query(max_length=50)])  Here the parameter q is of type str with a maximum length of 50. This information was communicated to FastAPI (or any other underlying library) using the Annotated keyword.
Question: I created a database with 3 tables using PostgreSQL and flask-sqlalchemy. I am querying 3 tables to get only their ids then I check their ids to see if there's any similar one then add the similar one to the third table but anytime i run it i get this error sqlalchemy.exc.ProgrammingError: (psycopg2.ProgrammingError) can't adapt type 'Row' [SQL: INSERT INTO login (student_id, name, timestamp) VALUES (%(student_id)s, %(name)s, %(timestamp)s)] [parameters: {'student_id': (1234567,), 'name': None, 'timestamp': datetime.datetime(2022, 4, 16, 21, 10, 53, 30512)}]  @app.route('/') def check(): id = Esp32.query.with_entities(Esp32.student_id).all() students = Student.query.with_entities(Student.student_id).all() logins = Login.query.with_entities(Login.student_id).all() for ids in id: if ids in students and ids not in logins: new = Login(student_id= ids) db.session.add(new) db.session.commit() return render_template('check.html', newlog = new)  please could someone tell me what this error means and why I am getting it

Answer: id is a query result. ids is a query row. To get one value from that row, you need to tell it which column (even if there is only one column): ids['student_id'].
Question: Trying to Read a external table from BigQuery but gettint a error  SCALA_VERSION="2.12" SPARK_VERSION="3.1.2" com.google.cloud.bigdataoss:gcs-connector:hadoop3-2.2.0, com.google.cloud.spark:spark-bigquery-with-dependencies_2.12:0.24.2' table = 'data-lake.dataset.member' df = spark.read.format('bigquery').load(table) df.printSchema()  Result: root |-- createdAtmetadata: date (nullable = true) |-- eventName: string (nullable = true) |-- producerName: string (nullable = true)  So when im print df.createOrReplaceTempView("member") spark.sql("select * from member limit 100").show()  i got this message error: INVALID_ARGUMENT: request failed: Only external tables with connections can be read with the Storage API.

Answer: As external tables are not supported in queries by spark, i tried the other way and got!  def read_query_bigquery(project, query): df = spark.read.format('bigquery') \ .option("parentProject", "{project}".format(project=project))\ .option('query', query)\ .option('viewsEnabled', 'true')\ .load() return df project = 'data-lake' query = 'select * from data-lake.dataset.member' spark.conf.set("materializationDataset",'dataset') df = read_query_bigquery(project, query) df.show()
Question: I am using AWS's MWAA service (2.2.2) to run a variety of DAGs, most of which are implemented with standard PythonOperator types. I bundle the DAGs into an S3 bucket alongside any shared requirements, then point MWAA to the relevant objects & versions. Everything runs smoothly so far. I would now like to implement a DAG using the PythonVirtualenvOperator type, which AWS acknowledge is not supported out of the box. I am following their guide on how to patch the behaviour using a custom plugin, but continue to receive an error from Airflow, shown at the top of the dashboard in big red writing:  DAG Import Errors (1) ... ... AirflowException: PythonVirtualenvOperator requires virtualenv, please install it.  I've confirmed that the plugin is indeed being picked up by Airflow (I see it referenced in the admin screen), and for the avoidance of doubt I am using the exact code provided by AWS in their examples for the DAG. AWS's documentation on this is pretty light and I've yet to stumble across any community discussion for the same. From AWS's docs, we'd expect the plugin to run at startup prior to any DAGs being processed. The plugin itself appears to effectively rewrite the venv command to use the pip-installed version, rather than that which is installed on the machine, however I've struggled to verify that things are happening in the order I expect. Any pointers on debugging the instance's behavior would be very much appreciated. Has anyone faced a similar issue? Is there a gap in the MWAA documentation that needs addressing? Am I missing something incredibly obvious? Possibly related, but I do see this warning in the scheduler's logs, which may indicate why MWAA is struggling to resolve the dependency?  WARNING: The script virtualenv is installed in '/usr/local/airflow/.local/bin' which is not on PATH. 

Answer: Airflow uses shutil.which to look for virtualenv. The installed virtualenv via requirements.txt isn't on the PATH. Adding the path to virtualenv to PATH solves this. The doc here is wrong https://docs.aws.amazon.com/mwaa/latest/userguide/samples-virtualenv.html import os from airflow.plugins_manager import AirflowPlugin import airflow.utils.python_virtualenv from typing import List def _generate_virtualenv_cmd(tmp_dir: str, python_bin: str, system_site_packages: bool) -> List[str]: cmd = ['python3','/usr/local/airflow/.local/lib/python3.7/site-packages/virtualenv', tmp_dir] if system_site_packages: cmd.append('--system-site-packages') if python_bin is not None: cmd.append(f'--python={python_bin}') return cmd airflow.utils.python_virtualenv._generate_virtualenv_cmd=_generate_virtualenv_cmd #This is the added path code os.environ["PATH"] = f"/usr/local/airflow/.local/bin:{os.environ['PATH']}" class VirtualPythonPlugin(AirflowPlugin): name = 'virtual_python_plugin'
Question: I have millions of sequences in fasta format and want to extract CDRs (CDR1, CDR2 and CDR3).I chose only one sequence as an example and tried to extract CDR1 but not able to extract CDR1. sequence:-'FYSHSAVTLDESGGGLQTPGGGLSLVCKASGFTFSSYGMMWVRQAPGKGLEYVAGIRNDA GDKRYGSAVQGRATISRDNGQSTVRLQLNNLRAEDTGTYFCAKESGCYWDSTHCIDAWGH GTEVIVSTGG'. cdr1 starts from:- 'VCKASGFTFS', with maximum three replacements but C at 2nd place is must. cdr1 ends at:-'WVRQAP', with maximum two replacements but R at 3rd place is must. Extracted cdr1 should be SYGMM def cdr1_in(cdr_in): #VCKASGFTFS pin=0 max_pin=3 if cdr[1]!='C': pin+=1 if cdr[0]!='V': pin+=1 if cdr[2]!='K': pin+=1 if cdr[3]!='A': pin+=1 if cdr[4]!='S': pin+=1 if cdr[5]!='G': pin+=1 if cdr[6]!='F': pin+=1 if cdr[7]!='T': pin+=1 if cdr[8]!='F': pin+=1 if cdr[9]!='S': pin+=1 if pin<max_pin: print('CDR_in pattern', cdr_in) # print('CDR_starts from', arr.index(cdr_in)+9) return (arr.index(cdr_in)+9) def cdr1_out(cdr_out):#WVRQAP pin=0 max_pin=2 if cdr[1]!='V': pin+=1 if cdr[0]!='W': pin+=1 if cdr[2]!='R': pin+=1 if cdr[3]!='Q': pin+=1 if cdr[4]!='A': pin+=1 if cdr[5]!='P': pin+=1 if pin<max_pin: # print('CDR_in pattern', cdr_out) # print('CDR_ends at', arr.index(cdr_out)) return (arr.index(cdr_out)) K=10 arr=sequence for i in range(len(arr)-k+1): slider=arr[i:k+i] print("CDR_1 is:", arr[cdr1_in(slider): cdr1_out(slider)]) 

Answer: I got it by the following method which works absolutely fine for me to find CDR1,2 and 3. All I need to define 3 different dictionaries having the definition ie prefix, suffix, max pin, fix position and pass them to the following code. Here I have performed this to find the CDR1, which gives me the desired output. dictionary_1={ 'cdr1_in': 'VCKASGFTFS', 'cdr1_out':'WVRQAP', 'max_pin_in':3, 'position_c':2, 'max_pin_out':2, 'position_r':3 } def cdr_out(dictionary_1,x): count=0 for i in range(len(x)-5): rider=x[i:i+6] # print(rider) if rider[2]=='R': # print(rider) for i in range(len(dictionary_1['cdr1_out'])): if rider[i]!=dictionary_1['cdr1_out'][i]: count+=1 if count<dictionary_1['max_pin_out']: # print(rider) return x.index(rider) def cdr_in(dictionary_1,x): count=0 for i in range(len(x)-9): rider=x[i:i+10] # print(rider) if rider[1]=='C': # print(rider) for i in range(len(dictionary_1['cdr1_in'])): if rider[i]!=dictionary_1['cdr1_in'][i]: count+=1 if count<dictionary_1['max_pin_in']: # print(rider) y=x.index(rider) z=cdr_out(dictionary_1,x) cdr=x[y+10:z] return cdr print(cdr_in(dictionary_1,x))  SYGMM
Question: I am trying to npm install for a project in my mac but for some reason it says python not found even though python3 command is working fine and I also set alias python to python3 in by ~/.zshrc and ~/.bash-profile and restarted several times but still the same issue. Screenshot of the issue.  NOTE: See comments for the solution

Answer: The problem is that Python is required in the system path to operate this command. Solutions:  Install pyenv Install either Python 2.7 or Python 3.x: pyenv install 2.7.18 or pyenv install 3.9.11 (for example) If you have more than one python version, ensure one of them is set as global: pyenv global 3.9.11 Add pyenv to your system path  On Mac, put this in your ~/.bashrc or ~/.zshrc: export PATH=$(pyenv root)/shims:$PATH, then run source ~/.bashrc or source ~/.zshrc For Windows, try npm config set python C:\Library\Python\Python310\python.exe (for example) via administrator   Run npm install again
Question: Whenever I run from firebase_admin import auth auth.verify_id_token(firebase_auth_token)  It throws the following error: Token used too early, 1650302066 < 1650302067. Check that your computer's clock is set correctly.  I'm aware that the underlying google auth APIs do check the time of the token, however as outlined here there should be a 10 second clock skew. Apparently, my server time is off by 1 second, however running this still fails even though this is well below the allowed 10 second skew. Is there a way to fix this?

Answer: This is how the firebase_admin.verify_id_token verifies the token: verified_claims = google.oauth2.id_token.verify_token( token, request=request, audience=self.project_id, certs_url=self.cert_url)  and this is the definition of google.oauth2.id_token.verify_token(...) def verify_token( id_token, request, audience=None, certs_url=_GOOGLE_OAUTH2_CERTS_URL, clock_skew_in_seconds=0, ):  As you can see, the function verify_token allows to specify a "clock_skew_in_seconds" but the firebase_admin function is not passing it along, thus the the default of 0 is used and since your server clock is off by 1 second, the check in verify_token fails. I would consider this a bug in firebase_admin.verify_id_token and maybe you can open an issue against the firebase admin SDK, but other than that you can only make sure, your clock is either exact or shows a time EARLIER than the actual time Edit: I actually opened an issue on GitHub for firebase/firebase-admin-Python and created an according pull request since I looked at all the source files already anyway... If and when the pull request is merged, the server's clock is allowed to be off by up to a minute.
Question: Could not find a working python interpreter. Please make sure one of the following is in your PATH: python python3 python3.8 python3.7 python2.7 python2 I installed python 3.10.4 Path is set in environment variables. Still not working.

Answer: How to set path:  Find the path to install Python on your computer. To do this, open the Windows search bar and type python.exe. Select the Open file location option. Copy path of python folder. To add Python To PATH In User Variables: Open My Computer\Properties\Advanced system settings\Advanced Environment Variables\Environment Variables. In the User Variables menu, find a variable named Path. Then paste the path you copied earlier into the Variable Value option using Ctrl+v and click OK. if you cannot find this variable, you may need to create it. To do this, click New. Then, in the variable name form, enter the path and paste your Python path into the variable value field. 6.You can also add Python to the PATH system variable. Although this is just an alternative and not needed if you have already added it to the Users variables.  To use the System Variables option, follow the steps highlighted above to copy the Python path and its script. Then go back to environment variables. Then, in the system variables segment, look for a variable named Path. Click this variable and click Edit.
Question: When I'm using the pandas.to_latex function to create latex table, the column names are unfortunately not bold. What can I do to make it bold?

Answer: Update I have been told on GitHub that this is allready possible with plain pandas but there is some missing documentation, which will be updated soon. You can use the line below. result = df.style.applymap_index( lambda v: "font-weight: bold;", axis="columns" ).to_latex(convert_css=True)  Old answer Here is a complete example, this is adapted from the offical documentation. There is a keyword to print bold columns bold_rows=True. Sadly there isn't a kyword parameter to do the same for the columns. But I can use this to check if my code gives the same result for the column headers. I use the result of to_latex() and split it in three sections. One section is the line with the column names. In this line I use a regular expression to add the \text{}-string. My code works only if you colum names don't have a whitespace. import pandas as pd df = pd.DataFrame( dict(name=['Raphael', 'Donatello'], mask=['red', 'purple'], weapon=['sai', 'bo staff'] ) ) ans = df.to_latex(bold_rows=True) split_middle = '\n\midrule\n' split_top = '\\toprule\n' top, mid = ans.split(split_middle) start, columns = top.split(split_top) columns = re.sub('(\w)+', '\\\\textbf{\g<0>}', columns) result = split_middle.join([split_top.join([start, columns]), mid]) >>> result \begin{tabular}{llll} \toprule {} & \textbf{name} & \textbf{mask} & \textbf{weapon} \\ \midrule \textbf{0} & Raphael & red & sai \\ \textbf{1} & Donatello & purple & bo staff \\ \bottomrule \end{tabular}  In the output you can see, that the header now is bold.
Question: I have something akin to the following piece of python code: import platform if platform.system() == "Windows": import winreg import win32api def do_cross_platform_thing() -> None: if platform.system() == "Windows": # do some overly complicated windows specific thing with winreg and win32api else: # do something reasonable for everyone else  Now, on linux, mypy complains that  it's missing imports because win32api doesn't exist, "module has no attribute ..." because the winreg module is defined, but basically disabled (all code is behind an 'is windows' check).  Is there any reasonable way to deal with this ? My current solution is  spamming # type: ignore everywhere --ignore-missing-imports  Are there any better solutions for this?

Answer: Ok, after checking the Docs as @SUTerliakov so kindly suggested, it seems that i have to change my if platform.system() == "Windows"  to this, semantically identical check: if sys.platform == "win32"  Only this second version triggers some magic builtin mypy special case that identifies this as a platform check and ignores the branch not applicable to its plattform.
Question: I'm trying to superimpose an image in the back of a matplotlib plot. It is being rendered as HTML in a flask website so I am saving the plot as an image before inserting it. The plot without the background image looks like this:  The code that produces the above output is here:  fname = 'scatter_averages.png' url_full = os.path.join(_path_full, fname) image = plt.imread("app/static/images/quantum_grid.jpg") if os.path.isfile(url_full): os.remove(url_full) plt.clf() df = self.text_numeric_averages() if df is not None: plt.figure(figsize=(6, 8)) fig, ax = plt.subplots() df.index += 1 x, y = df.iloc[:, 2], df.iloc[:, 1] ax.plot(x, y, '-o') plt.xlabel(df.columns[2]) plt.ylabel(df.columns[1]) for i in range(len(df)): xyi = df.iloc[i, :].values ax.annotate(str(df.index[i]) + " " + xyi[0][:3], (xyi[2], xyi[1])) axes = plt.gca() y_min, y_max = axes.get_ylim() x_min, x_max = axes.get_xlim() # ax.imshow(image, extent=[x_min, x_max, y_min, y_max]) plt.savefig(url_full)  The commented out line above is my attempt to get the image to superimpose. The output when that line is uncommented is this:  How do I keep the sizing and scale of the first image but use the background image in the second plot as the background? I'm not concerned with the image looking distorted.

Answer: ax.imshow(image, extent=[x_min, x_max, y_min, y_max], aspect="auto")  This will fix it.
Question: I find the new version pip(package installer for Python) has a colorful progress bar to show the downloading progress. How can I do that? Like this: 

Answer: pip itself is using the rich package! In particular, their progress bar docs show this example: from rich.progress import track for n in track(range(n), description="Processing..."): do_work(n)
Question: I want to test some function, that work with asyncpg. If I run one test at a time, it works fine. But if I run several tests at a time, all tests except the first one crash with the error asyncpg.exceptions._base.InterfaceError: cannot perform operation: another operation is in progress. Tests: @pytest.mark.asyncio async def test_project_connection(superuser_id, project_id): data = element_data_random(project_id) element_id = (await resolve_element_create(data=data, user_id=superuser_id))["id"] project_elements = (await db_projects_element_ids_get([project_id]))[project_id] assert element_id in project_elements @pytest.mark.asyncio async def test_project_does_not_exist(superuser_id): data = element_data_random(str(uuid.uuid4())) with pytest.raises(ObjectWithIdDoesNotExistError): await resolve_element_create(data=data, user_id=superuser_id)  All functions for work with db use pool look like: async def <some_db_func>(*args): pool = await get_pool() await pool.execute(...) # or fetch/fetchrow/fetchval  How I get the pool: db_pool = None async def get_pool(): global db_pool async def init(con): await con.set_type_codec('jsonb', encoder=ujson.dumps, decoder=ujson.loads, schema='pg_catalog') await con.set_type_codec('json', encoder=ujson.dumps, decoder=ujson.loads, schema='pg_catalog') if not db_pool: dockerfiles_dir = os.path.join(src_dir, 'dockerfiles') env_path = os.path.join(dockerfiles_dir, 'dev.env') try: # When code and DB inside docker containers host = 'postgres-docker' socket.gethostbyname(host) except socket.error: # When code on localhost, but DB inside docker container host = 'localhost' load_dotenv(dotenv_path=env_path) db_pool = await asyncpg.create_pool( database=os.getenv("POSTGRES_DBNAME"), user=os.getenv("POSTGRES_USER"), password=os.getenv("POSTGRES_PASSWORD"), host=host, init=init ) return db_pool  As far as I understand under the hood, asynсpg creates a new connection and runs the request inside that connection if you run the request through pool. Which makes it clear that each request should have its own connection. However, this error occurs, which is caused when one connection tries to handle two requests at the same time

Answer: Okay, thanks to @Adelin I realized that I need to run each asynchronous test synchronously. I I'm new to asyncio so I didn't understand it right away and found a solution. It was: @pytest.mark.asyncio async def test_...(*args): result = await <some_async_func> assert result == excepted_result  It become: def test_...(*args): async def inner() result = await <some_async_func> assert result == excepted_result asyncio.get_event_loop().run_until_complete(inner())
Question: I'm trying to add contacts to Sendgrid from a db which occasionally is storing the user email in punycode example-email@xn--yaho-sqa.com which translates to example-email@yahóo.com in Unicode. Anyway if I try and add the ascii version there's an error because sendgrid doesn't accept it - however it does accept the Unicode version. So is there a way to convert them in python. So I think long story short is there a way to decode punycode to Unicode? Edit As suggested in comments i tried 'example-email@yahóo.com'.encode('punycode').decode() which returns example-email@yaho.com-cgc so this is incorrect outside of python so is not a valid solution. Thanks in advance.

Answer: There is the xn-- ACE prefix in your encoded e-mail address:  The ACE prefix for IDNA is "xn--" or any capitalization thereof.  So apply the idna encoding (see Python Specific Encodings):  codec idna Implement RFC 3490, see also encodings.idna. Only errors='strict' is supported.  Result: 'yahóo.com'.encode('idna').decode() # 'xn--yaho-sqa.com'  and vice versa: 'xn--yaho-sqa.com'.encode().decode('idna') # 'yahóo.com'  You could use the idna library instead:  Support for the Internationalised Domain Names in Applications (IDNA) protocol as specified in RFC 5891. This is the latest version of the protocol and is sometimes referred to as “IDNA 2008”. This library also provides support for Unicode Technical Standard 46, Unicode IDNA Compatibility Processing. This acts as a suitable replacement for the “encodings.idna” module that comes with the Python standard library, but which only supports the older superseded IDNA specification (RFC 3490).
Question: If I have a Python module like this: from abc import ABC, abstractmethod class AbstractClass(ABC): @abstractmethod def method(self): pass class ConcreteClass1(AbstractClass): def method(self): print("hello") class ConcreteClass2(AbstractClass): def method(self): print("hello") class ConcreteClass3(AbstractClass): def method(self): print("hello") classes = [ ConcreteClass1, ConcreteClass2, ConcreteClass3, ] for c in classes: c().method()  And I hit it with mypy test.py, I get this: test.py:27: error: Cannot instantiate abstract class "AbstractClass" with abstract attribute "method"  The code runs without any issues though, and I can't see any issues in the logic. At no point am I trying to instantiate the AbstractClass directly. Some weird behavior I've noticed: If, instead of a loop, I do this: ... ConcreteClass1().method() ConcreteClass2().method() ConcreteClass3().method()  mypy is happy. Also, if, instead of 3 classes in the loop, I do 2: classes = [ ConcreteClass1, ConcreteClass2, #ConcreteClass3, ] for c in classes: c().method()  mypy is happy with that as well. What's going on? Is this a mypy bug? If so, can I tell mypy to ignore this "problem"?

Answer: The problem appears similar to mypy issues with abstract classes and dictionaries - for some reason, mypy can't typecheck this properly without a type annotation on the list: classes: list[Type[AbstractClass]] = [ ConcreteClass1, ConcreteClass2, ConcreteClass3, ]  (Change list to List if you're on Python 3.8 or below) You might want to file an issue for this bug on the mypy Github.