eshangj/stackoverflow_q_and_a_sample huggingface dataset:
Question: What is the best way to convert a sqlalchemy model to a pydantic schema (model) if it includes an enum field? Sqlalchemy import enum from sqlalchemy import Enum, Column, String from sqlalchemy.orm import declarative_base Base = declarative_base() class StateEnum(enum.Enum): CREATED = 'CREATED' UPDATED = 'UPDATED' class Adapter(Base): __tablename__ = 'adapters' id = Column(String, primary_key=True) friendly_name = Column(String(256), nullable=False) state: StateEnum = Column(Enum(StateEnum))  Pydantic from pydantic import BaseModel from enum import Enum class StateEnumDTO(str, Enum): CREATED = 'CREATED' UPDATED = 'UPDATED' class AdapterDTO(BaseModel): friendly_name: str state: StateEnumDTO # This currently cannot be converted? class Config: allow_population_by_field_name = True orm_mode = True use_enum_values = True  Conversion AdapterDTO.from_orm(Adapter(friendly_name='test', state=StateEnum.CREATED))  This leads to the error value is not a valid enumeration member; permitted: 'CREATED', 'UPDATED' (type=type_error.enum; enum_values=[<StateEnumDTO.CREATED: 'CREATED'>, <StateEnumDTO.UPDATED: 'UPDATED'>]) How can I configure either a.) the serialization with the from_orm method? or b.) the creation of the state field? c.) How to convert it the other way around? Is there a native way to do this with pydantic or how is this typically done? Update: Test case def test_enum_conversion_to_dto(): adapter = Adapter(id='1', friendly_name='test', state=StateEnum.CREATED) adapter_dto = AdapterDTO.from_orm(adapter) assert adapter_dto.state == StateEnumDTO.CREATED assert adapter_dto.state.value == StateEnum.CREATED.value 

Answer: Pydantic requires that both enum classes have the same type definition. In your case, StateEnum inherits from enum.Enum, but StateEnumDTO inherits from both str and enum.Enum. You can fix this issue by changing your SQLAlchemy enum definition: class StateEnum(str, enum.Enum): CREATED = 'CREATED' UPDATED = 'UPDATED'
Question: Suppose I want to plot a histogram of the same data twice: import matplotlib.pyplot as plt fig = plt.figure(figsize=(8,6)) ax1,ax2 = fig.subplots(nrows=2,ncols=1) ax1.hist(foo) ax2.hist(foo) ax2.set_yscale("log") ax2.set_xlabel("foo") fig.show()  Note that I call Axes.hist twice, and it could be expensive. I wonder if there is an easy way to re-use the return value of the 1st call to make the second one cheap.

Answer: In the ax.hist docs, there is a related example of reusing np.histogram output:  The weights parameter can be used to draw a histogram of data that has already been binned by treating each bin as a single point with a weight equal to its count. counts, bins = np.histogram(data) plt.hist(bins[:-1], bins, weights=counts)   We can use the same approach with ax.hist since it also returns counts and bins (along with a bar container): x = np.random.default_rng(123).integers(10, size=100) fig, (ax1, ax2) = plt.subplots(1, 2, figsize=(12, 3)) counts, bins, bars = ax1.hist(x) # original hist ax2.hist(bins[:-1], bins, weights=counts) # rebuilt via weights params    Alternatively, reconstruct the original histogram using ax.bar and restyle the width/alignment to match ax.hist: fig, (ax1, ax2) = plt.subplots(1, 2, figsize=(12, 3)) counts, bins, bars = ax1.hist(x) # original hist ax2.bar(bins[:-1], counts, width=1.0, align='edge') # rebuilt via ax.bar
Question: This is a sister question to How to set DEFAULT ON UPDATE CURRENT_TIMESTAMP in mysql with sqlalchemy?, but focused on Postgres instead of MySQL. Say we want to create a table users with a column datemodified that updates by default to the current timestamp whenever a row is updated. The solution given in the sister PR for MySQL is: user = Table( "users", Metadata, Column( "datemodified", TIMESTAMP, server_default=text("CURRENT_TIMESTAMP ON UPDATE CURRENT_TIMESTAMP"), ), )  How can I get the same functionality with a Postgres backend?

Answer: Eventually I implemented this using triggers as suggested by a_horse_with_no_name in the comments. Full SQLAlchemy implementation and integration with Alembic follow. SQLAlchemy implementation # models.py class User(Base): __tablename__ = "user" id = Column(Integer, primary_key=True) name = Column(Text) created_at = Column(DateTime, server_default=sqlalchemy.func.now(), nullable=False) updated_at = Column(DateTime)  # your_application_code.py import sqlalchemy as sa create_refresh_updated_at_func = """ CREATE FUNCTION {schema}.refresh_updated_at() RETURNS TRIGGER LANGUAGE plpgsql AS $func$ BEGIN NEW.updated_at := now(); RETURN NEW; END $func$; """ create_trigger = """ CREATE TRIGGER trig_{table}_updated BEFORE UPDATE ON {schema}.{table} FOR EACH ROW EXECUTE PROCEDURE {schema}.refresh_updated_at(); """ my_schema = "foo" engine.execute(sa.text(create_refresh_updated_at_func.format(schema=my_schema))) engine.execute(sa.text(create_trigger.format(schema=my_schema, table="user")))  Integration with Alembic In my case it was important to integrate the trigger creation with Alembic, and to add the trigger to n dimension tables (all of them having an updated_at column). # alembic/versions/your_version.py import sqlalchemy as sa create_refresh_updated_at_func = """ CREATE FUNCTION {schema}.refresh_updated_at() RETURNS TRIGGER LANGUAGE plpgsql AS $func$ BEGIN NEW.updated_at := now(); RETURN NEW; END $func$; """ create_trigger = """ CREATE TRIGGER trig_{table}_updated BEFORE UPDATE ON {schema}.{table} FOR EACH ROW EXECUTE PROCEDURE {schema}.refresh_updated_at(); """ def upgrade(): op.create_table(..., schema="foo") ... # Add updated_at triggers for all tables op.execute(sa.text(create_refresh_updated_at_func.format(schema="foo"))) for table in MY_LIST_OF_TABLES: op.execute(sa.text(create_trigger.format(schema="foo", table=table))) def downgrade(): op.drop_table(..., schema="foo") ... op.execute(sa.text("DROP FUNCTION foo.refresh_updated_at() CASCADE"))
Question: New poetry install fails on macOS 11.6/Xcode 13, using official installer: u@s-MacBook-Pro ~ % curl -sSL https://install.python-poetry.org | python3 - Retrieving Poetry metadata # Welcome to Poetry! This will download and install the latest version of Poetry, a dependency and package manager for Python. It will add the `poetry` command to Poetry's bin directory, located at: /Users/u/Library/Python/3.8/bin You can uninstall at any time by executing this script with the --uninstall option, and these changes will be reverted. Installing Poetry (1.1.12): An error occurred. Removing partial environment. Poetry installation failed. See /Users/u/poetry-installer-error-h3v75uw9.log for error logs. u@s-MacBook-Pro ~ % more /Users/u/poetry-installer-error-h3v75uw9.log dyld: Library not loaded: @executable_path/../Python3 Referenced from: /Users/u/Library/Application Support/pypoetry/venv/bin/python3 Reason: image not found Traceback: File "<stdin>", line 877, in main File "<stdin>", line 514, in run u@s-MacBook-Pro ~ %  There is no previous poetry install or reference to poetry in caches or Application Support: u@s-MacBook-Pro ~ % ls -al Library/Application\ Support/pypoetry total 0 drwxr-xr-x 2 u staff 64 26 Jan 16:58 . drwx------+ 55 u staff 1760 26 Jan 16:43 .. u@s-MacBook-Pro ~ % ls -al Library/Caches/pypoetry ls: Library/Caches/pypoetry: No such file or directory  Python version 3.8.9: u@s-MacBook-Pro ~ % python3 --version Python 3.8.9 u@s-MacBook-Pro ~ %  Xcode version: u@s-MacBook-Pro ~ % xcodebuild -version Xcode 13.2.1 Build version 13C100 u@s-MacBook-Pro ~ % 

Answer: I worked through this by installing a non-system version of python3, then installing poetry.  brew install pyenv pyenv install 3.8.12 pyenv local 3.8.12 curl -sSL https://install.python-poetry.org | python3 -
Question: Python ints are objects that encapsulate the actual number value. Can we mess with that value, for example setting the value of the object 1 to 2? So that 1 == 2 becomes True?

Answer: Yes, we can. But don't do this at home. Seriously, the 1 object is used in many places and I have no clue what this might break and what that might do to your computer. I reject all responsibility. But I found it interesting to learn about these things. The id function gives us the memory address and the ctypes module lets us mess with memory: import ctypes ctypes.memmove(id(1) + 24, id(2) + 24, 4) print(1 == 2) x = 40 print(x + 1)  Output: True 42  Try it online!. I tried it there because such sites have got to be protected from our hacking anyway.  More explanation / analysis: The memmove copied the value from the 2 object into the 1 object. Their size is 28 bytes each, but I skipped the first 24 bytes, because that's the object's reference count, type address, and value size, as we can view/verify as well: import ctypes, struct, sys x = 1 data = ctypes.string_at(id(x), 28) ref_count, type_address, number_of_digits, lowest_digit = \ struct.unpack('qqqi', data) print('reference count: ', ref_count, sys.getrefcount(x)) print('type address: ', type_address, id(type(x))) print('number of digits:', number_of_digits, -(-x.bit_length() // 30)) print('lowest digit: ', lowest_digit, x % 2**30)  Output (Try it online!): reference count: 135 138 type address: 140259718753696 140259718753696 number of digits: 1 1 lowest digit: 1 1  The reference count gets increased by the getrefcount call, but I don't know why by 3. Anyway, ~134 things other than us reference the 1 object, and we're potentially messing all of them up, so... really don't try this at home. The "digits" refer to how CPython stores ints as digits in base 230. For example, x = 2 ** 3000 has 101 such digits. Output for x = 123 ** 456 for a better test: reference count: 1 2 type address: 140078560107936 140078560107936 number of digits: 106 106 lowest digit: 970169057 970169057
Question: I have a dockerfile that currently only installs pip-tools FROM python:3.9 RUN pip install --upgrade pip && \ pip install pip-tools COPY ./ /root/project WORKDIR /root/project ENTRYPOINT ["tail", "-f", "/dev/null"]  I build and open a shell in the container using the following commands: docker build -t brunoapi_image . docker run --rm -ti --name brunoapi_container --entrypoint bash brunoapi_image  Then, when I try to run pip-compile inside the container I get this very weird error (full traceback): root@727f1f38f095:~/project# pip-compile Traceback (most recent call last): File "/usr/local/bin/pip-compile", line 8, in <module> sys.exit(cli()) File "/usr/local/lib/python3.9/site-packages/click/core.py", line 1128, in __call__ return self.main(*args, **kwargs) File "/usr/local/lib/python3.9/site-packages/click/core.py", line 1053, in main rv = self.invoke(ctx) File "/usr/local/lib/python3.9/site-packages/click/core.py", line 1395, in invoke return ctx.invoke(self.callback, **ctx.params) File "/usr/local/lib/python3.9/site-packages/click/core.py", line 754, in invoke return __callback(*args, **kwargs) File "/usr/local/lib/python3.9/site-packages/click/decorators.py", line 26, in new_func return f(get_current_context(), *args, **kwargs) File "/usr/local/lib/python3.9/site-packages/piptools/scripts/compile.py", line 342, in cli repository = PyPIRepository(pip_args, cache_dir=cache_dir) File "/usr/local/lib/python3.9/site-packages/piptools/repositories/pypi.py", line 106, in __init__ self._setup_logging() File "/usr/local/lib/python3.9/site-packages/piptools/repositories/pypi.py", line 455, in _setup_logging assert isinstance(handler, logging.StreamHandler) AssertionError  I have no clue what's going on and I've never seen this error before. Can anyone shed some light into this? Running on macOS Monterey

Answer: It is a bug, you can downgrade using: pip install "pip<22" https://github.com/jazzband/pip-tools/issues/1558
Question: We have a number of dataclasses representing various results with common ancestor Result. Each result then provides its data using its own subclass of ResultData. But we have trouble to annotate the case properly. We came up with following solution: from dataclasses import dataclass from typing import ClassVar, Generic, Optional, Sequence, Type, TypeVar class ResultData: ... T = TypeVar('T', bound=ResultData) @dataclass class Result(Generic[T]): _data_cls: ClassVar[Type[T]] data: Sequence[T] @classmethod def parse(cls, ...) -> T: self = cls() self.data = [self._data_cls.parse(...)] return self class FooResultData(ResultData): ... class FooResult(Result): _data_cls = FooResultData  but it stopped working lately with mypy error ClassVar cannot contain type variables [misc]. It is also against PEP 526, see https://www.python.org/dev/peps/pep-0526/#class-and-instance-variable-annotations, which we missed earlier. Is there a way to annotate this case properly?

Answer: At the end I just replaced the variable in _data_cls annotation with the base class and fixed the annotation of subclasses as noted by @rv.kvetch in his answer. The downside is the need to define the result class twice in every subclass, but in my opinion it is more legible than extracting the class in property. The complete solution: from dataclasses import dataclass from typing import ClassVar, Generic, Optional, Sequence, Type, TypeVar class ResultData: ... T = TypeVar('T', bound=ResultData) @dataclass class Result(Generic[T]): _data_cls: ClassVar[Type[ResultData]] # Fixed annotation here data: Sequence[T] @classmethod def parse(cls, ...) -> T: self = cls() self.data = [self._data_cls.parse(...)] return self class FooResultData(ResultData): ... class FooResult(Result[FooResultData]): # Fixed annotation here _data_cls = FooResultData
Question: I need to make a old code base compatible with Python3. The code needs to support Python2.7 and Python3 for some months. I would like to add this in very file: # -*- coding: utf-8 -*- from __future__ import absolute_import from __future__ import division from __future__ import print_function from __future__ import unicode_literals  I tried this: futurize --both-stages --unicode-literals --write --nobackups .  But this adds the unicode-literals future import only. Not the other future imports. I would like to avoid to write my own script which adds this, since adding this blindly does not work, since some files already have this header.

Answer: From the docs:  Only those __future__ imports deemed necessary will be added unless the --all-imports command-line option is passed to futurize, in which case they are all added.  If you want futurize to add all those imports unconditionally, you need to pass it the --all-imports flag.
Question: The list below provides examples of items and services that should not be billed separately. Please note that the list is not all inclusive. 1. Surgical rooms and services – To include surgical suites, major and minor, treatment rooms, endoscopy labs, cardiac cath labs, X-ray. 2. Facility Basic Charges - pulmonary and cardiology procedural rooms. The hospital’s charge for surgical suites and services shall include the entire above listed nursing personnel services, supplies, and equipment I want output like:  Surgical rooms and services Facility Basic Charges  there is first sentence also bold but we need to omit that sentence, we need to extract only those text which are represented with numbers

Answer: Use This Code: import pdfplumber import re demo = [] with pdfplumber.open('HCSC IL Inpatient_Outpatient Unbundling Policy- Facility.pdf') as pdf: for i in range(0, 50): try: text = pdf.pages[i] clean_text = text.filter(lambda obj: obj["object_type"] == "char" and "Bold" in obj["fontname"]) demo.append(str(re.findall(r'(\d+\.\s.*\n?)+', clean_text.extract_text())).replace('[]', ' ')) except IndexError: print("") break
Question: Trying to do Python (3.8.8) with type hinting and getting errors from mypy (0.931) that I can't really understand. import xml.etree.ElementTree as ET tree = ET.parse('plant_catalog.xml') # read in file and parse as XML root = tree.getroot() # get root node for plant in root: # loop through children if plant.find("LIGHT") and plant.find("LIGHT").text == "sun" print("foo")  This raises the mypy error Item "None" of "Optional[Element]" has no attribute "text". But why? I do check for the possibility of plant.find("LIGHT") returning None in the first half of the if clause. The second part accessing the .text attribute isn't even executed if the first part fails. If I modify to  lights = plant.find("LIGHT") if lights: if lights.text == selection: print("foo")  the error is gone. So is this because the plant object might still change in between the first check and the second? But assigning to a variable doesn't automatically copy the content, its still just a reference to an object that might change. So why does it pass the second time? (Yes, I know that repeating the .find() twice is also not time-efficient.)

Answer: mypy doesn't know that plant.find("LIGHT") always returns the same value, so it doesn't know that your test is a proper guard. So you need to assign it to a variable. As far as mypy is concerned, the variable can't change from one object to another without being reassigned, and its contents can't change if you don't perform some other operation on it.
Question: I have a dataframe that looks like this import pandas as pd import numpy as np # Create data set. dataSet = {'id': ['A', 'A', 'B'], 'id_2': [1, 2, 1] , 'number': [320, 169, 120], 'add_number' : [4,6,3]} # Create dataframe with data set and named columns. df = pd.DataFrame(dataSet, columns= ['id', 'id_2','number', 'add_number']) id id_2 number add_number 0 A 1 320 4 1 A 2 169 6 2 B 1 120 3  I would like use number and add_number so that I can explode this dynamically, ie) 320 + 4 would have [320,321,322,323,324] (up to 324, and would like to explode on this) DESIRED OUTPUT  id id_2 number 0 A 1 320 1 A 1 321 2 A 1 322 3 A 1 323 4 A 1 324 5 A 2 169 6 A 2 170 7 A 2 171 8 A 2 172 9 A 2 173 10 A 2 174 11 A 2 175 12 B 1 120 13 B 1 121 14 B 1 122 15 B 1 123  I looked over explode, wide_to_long pandas function, but I do not know where to start, any sense of direction would be appreciated!!

Answer: You try using np.arange and explode: df['range'] = df.apply(lambda x: np.arange(x['number'], x['number']+x['add_number']+1), axis=1) df.explode('range')  or df['range'] = [np.arange(n, n+a+1) for n, a in zip(df['number'],df['add_number'])] df.explode('range')  Output:  id id_2 number add_number range 0 A 1 320 4 320 0 A 1 320 4 321 0 A 1 320 4 322 0 A 1 320 4 323 0 A 1 320 4 324 1 A 2 169 6 169 1 A 2 169 6 170 1 A 2 169 6 171 1 A 2 169 6 172 1 A 2 169 6 173 1 A 2 169 6 174 1 A 2 169 6 175 2 B 1 120 3 120 2 B 1 120 3 121 2 B 1 120 3 122 2 B 1 120 3 123
Question: Let's say we have a pandas dataframe:  name age sal 0 Alex 20 100 1 Jane 15 200 2 John 25 300 3 Lsd 23 392 4 Mari 21 380  Let's say, a few rows are now deleted and we don't know the indexes that have been deleted. For example, we delete row index 1 using df.drop([1]). And now the data frame comes down to this:  fname age sal 0 Alex 20 100 2 John 25 300 3 Lsd 23 392 4 Mari 21 380  I would like to get the value from row index 3 and column "age". It should return 23. How do I do that? df.iloc[3, df.columns.get_loc('age')] does not work because it will return 21. I guess iloc takes the consecutive row index?

Answer: Use .loc to get rows by label and .iloc to get rows by position: >>> df.loc[3, 'age'] 23 >>> df.iloc[2, df.columns.get_loc('age')] 23  More about Indexing and selecting data
Question: I am trying to crop a pdf and save it to s3 with same name using lambda. I am getting error on the data type being a fitz.fitz.page import os import json import boto3 from urllib.parse import unquote_plus import fitz, sys from io import BytesIO OUTPUT_BUCKET_NAME = os.environ["OUTPUT_BUCKET_NAME"] OUTPUT_S3_PREFIX = os.environ["OUTPUT_S3_PREFIX"] SNS_TOPIC_ARN = os.environ["SNS_TOPIC_ARN"] SNS_ROLE_ARN = os.environ["SNS_ROLE_ARN"] def lambda_handler(event, context): textract = boto3.client("textract") if event: file_obj = event["Records"][0] bucketname = str(file_obj["s3"]["bucket"]["name"]) filename = unquote_plus(str(file_obj["s3"]["object"]["key"])) doc = fitz.open() s3 = boto3.resource('s3') obj = s3.Object(bucketname, filename) fs = obj.get()['Body'].read() pdf=fitz.open("pdf", stream=BytesIO(fs)) #pdf.close() rect=fitz.Rect(0.0, 0.0, 595.0, 842.0) #page = pdf[0] page1 = doc.new_page(width = rect.width, # new page with ... height = rect.height) page1.show_pdf_page(rect, pdf, 0) print(type(doc)) print(type(page1)) s3.Bucket(bucketname).put_object(Key=filename, Body=page1) 

Answer: This is happening because the page1 object is defined using fitz.fitz.page and the type expected by S3 put object is bytes. In order to solve the issue, you can use the write function of the new PDF (doc) and get the output of it which is in bytes format that you could pass to S3 then. # Save fil first. new_bytes = doc.write() s3.Bucket(bucketname).put_object(Key=filename, Body=new_bytes)
Question: I'm using python requests library, I need send a request without a user-agent header. I found this question, but it's for Urllib2. I'm trying to simulate an Android app which does this when calling a private API. I try to set User-Agent to None as in the following code, but it doesn't work. It still sends User-Agent: python-requests/2.27.1. Is there any way? headers = requests.utils.default_headers() headers['User-Agent'] = None requests.post(url, *args, headers=headers, **kwargs) 

Answer: The requests library is built on top of the urllib3 library. So, when you pass None User-Agent header to the requests's post method, the urllib3 set their own default User-Agent import requests r = requests.post("https://httpbin.org/post", headers={ "User-Agent": None, }) print(r.json()["headers"]["User-Agent"])  Output python-urllib3/1.26.7  Here the urllib3 source of connection.py class HTTPConnection(_HTTPConnection, object): ... def request(self, method, url, body=None, headers=None): if headers is None: headers = {} else: # Avoid modifying the headers passed into .request() headers = headers.copy() if "user-agent" not in (six.ensure_str(k.lower()) for k in headers): headers["User-Agent"] = _get_default_user_agent() super(HTTPConnection, self).request(method, url, body=body, headers=headers)  So, you can monkey patch it to disable default User-Agent header import requests from urllib3 import connection def request(self, method, url, body=None, headers=None): if headers is None: headers = {} else: # Avoid modifying the headers passed into .request() headers = headers.copy() super(connection.HTTPConnection, self).request(method, url, body=body, headers=headers) connection.HTTPConnection.request = request r = requests.post("https://httpbin.org/post", headers={ "User-Agent": None, }) print(r.json()["headers"])  Output { 'Accept': '*/*', 'Accept-Encoding': 'gzip, deflate', 'Content-Length': '0', 'Host': 'httpbin.org', 'X-Amzn-Trace-Id': 'Root=1-61f7b53b-26c4c8f6498c86a24ff05940' }  Also, consider to provide browser-like User-Agent like this Mozilla/5.0 (Macintosh; Intel Mac OS X 12_1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/97.0.4692.99 Safari/537.36. Maybe it solves your task with less effort