eshangj/stackoverflow_q_and_a_sample huggingface dataset:
Question: import asyncio from sqlalchemy import Column from sqlalchemy import DateTime from sqlalchemy import ForeignKey from sqlalchemy import func from sqlalchemy import Integer from sqlalchemy import String from sqlalchemy.ext.asyncio import AsyncSession from sqlalchemy.ext.asyncio import create_async_engine from sqlalchemy.future import select from sqlalchemy.orm import declarative_base from sqlalchemy.orm import relationship from sqlalchemy.orm import selectinload from sqlalchemy.orm import sessionmaker engine = create_async_engine( "postgresql+asyncpg://user:pass@localhost/db", echo=True, ) # expire_on_commit=False will prevent attributes from being expired # after commit. async_session = sessionmaker( engine, expire_on_commit=False, class_=AsyncSession ) Base = declarative_base() class A(Base): __tablename__ = "a" id = Column(Integer, primary_key=True) name = Column(String, unique=True) data = Column(String) create_date = Column(DateTime, server_default=func.now()) bs = relationship("B") # required in order to access columns with server defaults # or SQL expression defaults, subsequent to a flush, without # triggering an expired load __mapper_args__ = {"eager_defaults": True} class B(Base): __tablename__ = "b" id = Column(Integer, primary_key=True) a_id = Column(ForeignKey("a.id")) data = Column(String) async with engine.begin() as conn: await conn.run_sync(Base.metadata.drop_all) await conn.run_sync(Base.metadata.create_all) async with async_session() as session: async with session.begin(): session.add_all( [ A(bs=[B(), B()], data="a1"), A(bs=[B()], data="a2"), ] ) async with async_session() as session: result = await session.execute(select(A).order_by(A.id)) a1 = result.scalars().first() # no issue: print(a1.name, a1.data) # throws error: print(a1.bs)  Trying to access a1.bs gives this error:  59 current = greenlet.getcurrent() 60 if not isinstance(current, _AsyncIoGreenlet): ---> 61 raise exc.MissingGreenlet( 62 "greenlet_spawn has not been called; can't call await_() here. " 63 "Was IO attempted in an unexpected place?" MissingGreenlet: greenlet_spawn has not been called; can't call await_() here. Was IO attempted in an unexpected place? (Background on this error at: https://sqlalche.me/e/14/xd2s) 

Answer: This is how: from sqlalchemy.orm import selectinload async with async_session() as session: result = await session.execute(select(A).order_by(A.id) .options(selectinload(A.bs))) a = result.scalars().first() print(a.bs)  key is using the selectinload method to prevent implicit IO UPDATE There are a few alternatives to selectinload like joinedload, lazyload. I am still trying to understand the differences.
Question: When starting a Flask app with: $ flask run  I received the error:  Error: While importing 'wsgi', an ImportError was raised. Usage: flask [OPTIONS] COMMAND [ARGS]...` ...   However, there is no stack trace or other information provided. What is the best way to get the ImportError stack trace?

Answer: Import the Flask app at the Python interpreter prompt To see the ImportError stack trace, open a Python interpreter prompt and import the module that loads the Flask app (usually app.py or wsgi.py). If applicable, be sure that your virtual environment is activated. $ python >>> from my_app_folder import app  Set the FLASK_APP environment variable If you can import the Flask app module using the Python interpreter without error, try setting the FLASK_APP environment variable to point to the Flask app module. $ FLASK_APP='my_app_folder/app' FLASK_ENV=development flask run
Question: Am trying to implement TLS for mqtt and has followed the tutorials from the link below http://www.steves-internet-guide.com/mosquitto-tls/ I followed exactly how it has been instructed to generate certificates using openssl and pasted in the location of mqtt and changed the conf of mqtt and restarted the service. But when I try to connect to mqtt using tls it shows the below error message ssl.SSLCertVerificationError: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: self signed certificate (_ssl.c:1124)  And python code is client1 = paho.Client("control1") client1.tls_set(ca_certs="ca.crt") client1.tls_insecure_set(True) client1.connect("localhost", 8883) client1.loop_forever()  where ca.crt is in the project directory.

Answer: The message you are receiving indicates that the broker's server certificate is not trusted (because it is self-signed), therefore paho is not being correctly told it is trustworthy. It is possible your fake certificate authority's root certificate (the ca.crt file you feed to paho) is not properly signed or generated, or the certificates that Mosquitto is using are not signed correctly. Either way, you likely need to start the entire process over to be 100% certain everything was done right.  Generate the fake certificate authority's (CA) signing key $ openssl genrsa -des3 -out ca.key 2048  Generate a certificate signing request for the fake CA $ openssl req -new -key ca.key -out ca-cert-request.csr -sha256  Give the organization a name like "Fake Authority" and do not enter a common name (since your fake CA does not actually live on a server with a name)   Create the fake CA's root certificate $ openssl x509 -req -in ca-cert-request.csr -signkey ca.key -out ca-root-cert.crt -days 365 -sha256  Create the server / mqtt broker's keypair $ openssl genrsa -out server.key 2048  Create a certificate signing request using the server key to send to the fake CA for identity verification $ openssl req -new -key server.key -out server-cert-request.csr -sha256  Give the organization a name like "Localhost MQTT Broker Inc." and the common name should be localhost or the exact domain you use to connect to the mqtt broker   Now acting as the fake CA, you receive the server's request for your signature. You have verified the server is who it says it is (an MQTT broker operating on localhost), so create a new certificate & sign it with all the power of your fake authority. $ openssl x509 -req -in server-cert-request.csr -CA ca-root-cert.crt -CAkey ca.key -CAcreateserial -out server.crt -days 360   Now you have everything you need. Make sure (as in Steve's tutorial) Mosquitto is loading the following in mosquitto.conf: listener 8883 cafile certs\ca-root-cert.crt keyfile certs\server.key certfile certs\server.crt  Make sure paho-mqtt is loading the fake CA's root certificate. client1.tls_set(ca_certs="ca-root-cert.crt")  This is how it knows that mosquitto's server.crt is legitimately signed by a "real and trusted authority" and is not "self-signed" and thus untrusted. Mosquitto and paho should now be able to securely connect and communicate.
Question: I've been looking for the answer to this for a good solid week now, with no success. I've looked at every StackOverflow post, every article from Google and every related Github issue I could find. Most related errors seem to be older, so I'm wondering if my issue is slightly different due to me being on macOS Big Sur. The issue: When I try to run yarn install in my local repo, I receive an error related to node-gyp and a python executable that is unable to be found. Here is what my terminal shows: yarn install v1.22.17 ...other stuff [4/4] 🔨 Building fresh packages... [6/13] ⠐ node-sass [2/13] ⠐ node-sass [10/13] ⠐ metrohash [4/13] ⠐ fsevents error /Users/jimmiejackson/Documents/repositories/repo-name/node_modules/metrohash: Command failed. Exit code: 1 Command: node-gyp rebuild Arguments: Directory: /Users/jimmiejackson/Documents/repositories/repo-name/node_modules/metrohash Output: gyp info it worked if it ends with ok gyp info using node-gyp@3.8.0 gyp info using node@12.18.0 | darwin | x64 gyp ERR! configure error gyp ERR! stack Error: Can't find Python executable "/usr/local/opt/python@3.9/bin/python3", you can set the PYTHON env variable. gyp ERR! stack at PythonFinder.failNoPython (/Users/jimmiejackson/Documents/repositories/repo-name/node_modules/node-gyp/lib/configure.js:484:19) gyp ERR! stack at PythonFinder.<anonymous> (/Users/jimmiejackson/Documents/repositories/repo-name/node_modules/node-gyp/lib/configure.js:406:16) gyp ERR! stack at F (/Users/jimmiejackson/Documents/repositories/repo-name/node_modules/which/which.js:68:16) gyp ERR! stack at E (/Users/jimmiejackson/Documents/repositories/repo-name/node_modules/which/which.js:80:29) gyp ERR! stack at /Users/jimmiejackson/Documents/repositories/repo-name/node_modules/which/which.js:89:16 gyp ERR! stack at /Users/jimmiejackson/Documents/repositories/repo-name/node_modules/isexe/index.js:42:5 gyp ERR! stack at /Users/jimmiejackson/Documents/repositories/repo-name/node_modules/isexe/mode.js:8:5 gyp ERR! stack at FSReqCallback.oncomplete (fs.js:167:21) gyp ERR! System Darwin 20.6.0 gyp ERR! command "/Users/jimmiejackson/.nvm/versions/node/v12.18.0/bin/node" "/Users/jimmiejackson/Documents/repositories/repo-name/node_modules/metrohash/node_modules/.bin/node-gyp" "rebuild" gyp ERR! cwd /Users/jimmiejackson/Documents/repositories/repo-name/node_modules/metrohash  I'm not entirely sure what this error means or why this node module is searching for python3. I've tried running npm set config /path/to/python, downloading python3, setting the PYTHON path in my .zshrc profile, but nothing seems to be working. It's entirely possible that my lack of understanding of the issue means that I'm on the right path but didn't quite get something right. Any ideas?

Answer: Reading the gyp-node source might helps. Here are some steps you can try.  Install python2. You should make sure that in the terminal, which -a python2 only returns one python2 and python2 -V returns the correct 2.x version.  override PYTHON env. export PYTHON=python2.  Rerun the install.   If there's still an error, probably the error message is different.
Question: On PyCharm you can right-click a file -> Refactor -> Rename. And it would rename that file and update all import statements in all files in the project. In VS Code, while I can rename and refactor symbols by highlighting them -> F2, this only works for modules, classes, their members, and variables. E.g. I have an utils/__init__.py with: from utils.readers import CSVReader  utils/readers.py with: class CSVReader: pass  And main.py with: from utils import CSVReader r = CSVReader()  I would like to, for example, rename utils/readers.py -> utils/local_readers.py and have VS Code auto-update utils/__init__.py with: from utils.local_readers import CSVReader  Dozens of google search results point to Move TS (for TypeScript only). Is there a similar extension for Python, or some built-in hotkey I missed?

Answer: Check out the enhanced module re-naming coming to vscode v1.63 (see release notes: python module rename refactoring.  Module rename refactoring You can now more easily rename modules with the Python and Pylance extensions. Once you rename a Python module, you’ll be prompted to choose whether you’d like to change all import and references throughout your code. If you’re not sure, you can first preview what all the changes will look like before you make the decision. Once you’re confident about it, you can click on Apply Refactoring, or Discard Refactoring to not have them applied.
Question: I'm struggling to get the syntax to create a UUID field when creating a model in my FastAPI application. I'm using SQLModel. So basically, my models.py file looks like this: from datetime import datetime from typing import Optional import uuid from sqlalchemy import Column, DateTime from sqlalchemy.dialects import postgresql as psql from sqlmodel import SQLModel, Field class ModelBase(SQLModel): """ Base class for database models. """ id: Optional[int] = Field(default=None, primary_key=True) created_at: datetime = Field(sa_column=Column(DateTime(timezone=True), default=datetime.utcnow)) updated_at: datetime = Field(sa_column=Column(DateTime(timezone=True), onupdate=datetime.utcnow, default=datetime.utcnow)) class UUIDModelBase(ModelBase, table=True): """ Base class for UUID-based models. """ uuid: uuid.UUID = Field(sa_column=Column(psql.UUID(as_uuid=True)), default=uuid.uuid4)  The above errors out with AttributeError: 'FieldInfo' object has no attribute 'UUID'  I also tried id: uuid.UUID = Column(psql.UUID(as_uuid=True), default=uuid.uuid4) TypeError: Boolean value of this clause is not defined  Also uuid: uuid.UUID = Column(psql.UUID(as_uuid=True), default=uuid.uuid4) AttributeError: Neither 'Column' object nor 'Comparator' object has an attribute 'UUID'  and uuid: uuid.UUID = Field(default_factory=uuid.uuid4, index=True, nullable=False) AttributeError: 'FieldInfo' object has no attribute 'UUID'  You get the idea. The errors are not helping me, I just need the right syntax. In this case, I'm not actually looking to use UUID as a primary key. And as you can tell from the imports, I'm using postgreSQL. The database is based on a postgres:12 docker image.

Answer: The interpreter might be using UUID of your actual field uuid instead of the imported package. So, you can change the code as follows. import uuid as uuid_pkg from sqlalchemy import Field from sqlmodel import Field class UUIDModelBase(ModelBase): """ Base class for UUID-based models. """ uuid: uuid_pkg.UUID = Field( default_factory=uuid_pkg.uuid4, primary_key=True, index=True, nullable=False, )  Reference: https://github.com/tiangolo/sqlmodel/issues/140
Question: I want to train a custom dataset on using faster_rcnn or mask_rcnn with the Pytorch and Detectron2 .Everything works well but I wanted to know I want to know what are the results I have. [11/29 20:16:31 d2.utils.events]: eta: 0:24:04 iter: 19 total_loss: 9.6 loss_cls: 1.5 loss_box_reg: 0.001034 loss_mask: 0.6936 loss_rpn_cls: 6.773 loss_rpn_loc: 0.5983 time: 1.4664 data_time: 0.0702 lr: 4.9953e-06 max_mem: 2447M  I have this as result and I want to know what all of this means

Answer: Those are metrics printed out at every iteration of the training loop. The most important ones are the loss values, but below are basic descriptions of them all (eta and iter are self-explanatory I think). total_loss: This is a weighted sum of the following individual losses calculated during the iteration. By default, the weights are all one.  loss_cls: Classification loss in the ROI head. Measures the loss for box classification, i.e., how good the model is at labelling a predicted box with the correct class.  loss_box_reg: Localisation loss in the ROI head. Measures the loss for box localisation (predicted location vs true location).  loss_rpn_cls: Classification loss in the Region Proposal Network. Measures the "objectness" loss, i.e., how good the RPN is at labelling the anchor boxes as foreground or background.  loss_rpn_loc: Localisation loss in the Region Proposal Network. Measures the loss for localisation of the predicted regions in the RPN.  loss_mask: Mask loss in the Mask head. Measures how "correct" the predicted binary masks are. For more details on the losses (1) and (2), take a look at the Fast R-CNN paper and the code. For more details on the losses (3) and (4), take a look at the Faster R-CNN paper and the code. For more details on the loss (5), take a look at the Mask R-CNN paper and the code.   time: Time taken by the iteration. data_time: Time taken by the dataloader in that iteration. lr: The learning rate in that iteration. max_mem: Maximum GPU memory occupied by tensors in bytes.
Question: I use python and opencv-python to capture frames from video, then use ffmpeg command to push rtsp stream with pipe. I can play the rtsp stream via gstreamer and vlc. However, a display device cannot decode and play the rtsp-stream because it cannot receive SPS and PPS frames. Use wireshark to capture stream, found that it doesn't send sps and pps frames, only send IDR frames. The key codes are as follows. # ffmpeg command command = ['ffmpeg', '-re', '-y', '-f', 'rawvideo', '-vcodec', 'rawvideo', '-pix_fmt', 'bgr24', '-s', "{}x{}".format(width, height), '-r', str(fps), '-i', '-', '-c:v', 'libx264', '-preset', 'ultrafast', '-f', 'rtsp', '-flags', 'local_headers', '-rtsp_transport', 'tcp', '-muxdelay', '0.1', rtsp_url] p = sp.Popen(command, stdin=sp.PIPE) while (cap.isOpened()): ret, frame = cap.read() if not ret: cap = cv2.VideoCapture(video_path) continue p.stdin.write(frame.tobytes()  May be I miss some options of ffmpeg command?

Answer: Try adding the arguments '-bsf:v', 'dump_extra'. According to FFmpeg Bitstream Filters Documentation:  dump_extra Add extradata to the beginning of the filtered packets except when said packets already exactly begin with the extradata that is intended to be added.  The filter supposed to add SPS and PPS NAL units with every key frame.  Here is a complete code sample: import subprocess as sp import cv2 rtsp_url = 'rtsp://localhost:31415/live.stream' video_path = 'input.mp4' # We have to start the server up first, before the sending client (when using TCP). See: https://trac.ffmpeg.org/wiki/StreamingGuide#Pointtopointstreaming ffplay_process = sp.Popen(['ffplay', '-rtsp_flags', 'listen', rtsp_url]) # Use FFplay sub-process for receiving the RTSP video. cap = cv2.VideoCapture(video_path) width = int(cap.get(cv2.CAP_PROP_FRAME_WIDTH)) # Get video frames width height = int(cap.get(cv2.CAP_PROP_FRAME_HEIGHT)) # Get video frames height fps = int(cap.get(cv2.CAP_PROP_FPS)) # Get video framerate # FFmpeg command command = ['ffmpeg', '-re', '-y', '-f', 'rawvideo', '-vcodec', 'rawvideo', '-pix_fmt', 'bgr24', '-s', "{}x{}".format(width, height), '-r', str(fps), '-i', '-', '-c:v', 'libx264', '-preset', 'ultrafast', '-f', 'rtsp', #'-flags', 'local_headers', '-rtsp_transport', 'tcp', '-muxdelay', '0.1', '-bsf:v', 'dump_extra', rtsp_url] p = sp.Popen(command, stdin=sp.PIPE) while (cap.isOpened()): ret, frame = cap.read() if not ret: break p.stdin.write(frame.tobytes()) p.stdin.close() # Close stdin pipe p.wait() # Wait for FFmpeg sub-process to finish ffplay_process.kill() # Forcefully close FFplay sub-process   Notes:  '-flags', 'local_headers' are not valid arguments in my version of FFmpeg. I don't know how to verify my solution, so I could be wrong...
Question: I am trying to save a model to MLFlow, but as I have a custom prediction pipeline to retrieve data, I need to save extra metadata into the model. I tried using my custom signature class, which It does the job correctly and saves the model with the extra metadata in the MLModel file (YAML format). But when want to load the model from the MLFlow registry, the signature is not easy accesible. mlflow.sklearn.log_model(model, "model", signature = signature)  I've also tried to save an extra dictionary at the log_model function, but it saves it in the conda.yaml file: mlflow.sklearn.log_model(model, "model", {"metadata1":"value1", "metadata2":"value2"})  Should I make my own flavour? Or my own Model inheritance? I've seen here that the PyFuncModel recieves some metadata class and an implementation to solve this, but I don't know where should I pass my own implementations to PyFuncModel on an experiment script. Here's a minimal example: import mlflow import numpy as np import pandas as pd from sklearn.linear_model import LogisticRegression metadata_dic = {"metadata1": "value1", "metadata2": "value2"} X = np.array([[-2, -1, 0, 1, 2, 1],[-2, -1, 0, 1, 2, 1]]).T y = np.array([0, 0, 1, 1, 1, 0]) X = pd.DataFrame(X, columns=["X1", "X2"]) y = pd.DataFrame(y, columns=["y"]) model = LogisticRegression() model.fit(X, y) mlflow.sklearn.log_model(model, "model") 

Answer: Finally, I made a class that contains every metadata and saved it as an model argument: model = LogisticRegression() model.fit(X, y) model.metadata = ModelMetadata(**metadata_dic) mlflow.sklearn.log_model(model, "model")  Here I lost the customizable predict process, but after reading the MLFlow documentation is not very clear how to proceed. If anyone finds a good approach It would be very appreciated.
Question: I'm trying to load my pretrained model (yolov5n) and test it with the following code in PyTorch: import os import torch model = torch.load(os.getcwd()+'/weights/last.pt') # Images imgs = ['https://example.com/img.jpg'] # Inference results = model(imgs) # Results results.print() results.save() # or .show() results.xyxy[0] # img1 predictions (tensor) results.pandas().xyxy[0] # img1 predictions (pandas)  and I'm getting the following error:  ModuleNotFoundError Traceback (most recent call last) in 3 import torch 4 ----> 5 model = torch.load(os.getcwd()+'/weights/last.pt')  My model is located in the folder /weights/last.py, I'm not sure what I'm doing false. Could you please tell me, what it's missing in my code.

Answer: You should be able to find the weights in this directory: yolov5/runs/train/exp/weights/last.pt Then you load the weights with a line like this: model = torch.hub.load('ultralytics/yolov5', 'custom', path='yolov5/runs/train/exp/weights/last.pt', force_reload=True)  I have an example of a notebook that loads custom models and from that directory after training the model here https://github.com/pylabel-project/samples/blob/main/pylabeler.ipynb
Question: I have a python script where I'm using pandas for transformations/manipulation of my data. I know I have some "inefficient" blocks of code. My question is, if pyspark is supposed to be much faster, can I just replace these blocks using pyspark instead of pandas or do I need everything to be in pyspark? If I'm in Databricks, how much does this really matter since it's already on a spark cluster?

Answer: If the data is small enough that you can use pandas to process it, then you likely don't need pyspark. Spark is useful when you have such large data sizes that it doesn't fit into memory in one machine since it can perform distributed computation. That being said, if the computation is complex enough that it could benefit from a lot of parallelization, then you could see an efficiency boost using pyspark. I'm more comfortable with pyspark's APIs than pandas, so I might end up using pyspark anyways, but whether you'll see an efficiency boost depends a lot on the problem.
Question: I've created a small dataset comparing coffee drink prices per cup size. When I pivot my dataset the output automatically reorders the index (the 'Size' column) alphabetically. Is there a way to assign the different sizes a numerical level (e.g. small = 0, medium = 1, large = 2) and reorder the rows this way instead? I'm know this can be done in R using the forcats library (using fct_relevel for example), but I'm not aware of how to do this in python. I would prefer to keep the solution to using numpy and pandas. data = {'Item': np.repeat(['Latte', 'Americano', 'Cappuccino'], 3), 'Size': ['Small', 'Medium', 'Large']*3, 'Price': [2.25, 2.60, 2.85, 1.95, 2.25, 2.45, 2.65, 2.95, 3.25] } df = pd.DataFrame(data, columns = ['Item', 'Size', 'Price']) df = pd.pivot_table(df, index = ['Size'], columns = 'Item') df # Price # Item Americano Cappuccino Latte # Size # Large 2.45 3.25 2.85 # Medium 2.25 2.95 2.60 # Small 1.95 2.65 2.25 

Answer: You can use a Categorical type with ordered=True: df.index = pd.Categorical(df.index, categories=['Small', 'Medium', 'Large'], ordered=True) df = df.sort_index()  output:  Price Item Americano Cappuccino Latte Small 1.95 2.65 2.25 Medium 2.25 2.95 2.60 Large 2.45 3.25 2.85  You can access the codes with: >>> df.index.codes array([0, 1, 2], dtype=int8)  If this was a Series: >>> series.cat.codes