eshangj/stackoverflow_q_and_a_sample huggingface dataset:
Question: Concept: Using AWS Lambda functions with Python and Selenium, I want to create a undetectable headless chrome scraper by passing a headless chrome test. I check the undetectability of my headless scraper by opening up the test and taking a screenshot. I ran this test on a Local IDE and on a Lambda server.  Implementation: I will be using a python library called selenium-stealth and will follow their basic configuration: stealth(driver, languages=["en-US", "en"], vendor="Google Inc.", platform="Win32", webgl_vendor="Intel Inc.", renderer="Intel Iris OpenGL Engine", fix_hairline=True, )  I implemented this configuration on a Local IDE as well as an AWS Lambda Server to compare the results.  Local IDE: Found below are the test results running on a local IDE:   Lambda Server: When I run this on a Lambda server, both the WebGL Vendor and Renderer are blank. as shown below:  I even tried to manually change the WebGL Vendor/Renderer using the following JavaScript command: driver.execute_cdp_cmd('Page.addScriptToEvaluateOnNewDocument', {"source": "WebGLRenderingContext.prototype.getParameter = function(parameter) {if (parameter === 37445) {return 'VENDOR_INPUT';}if (parameter === 37446) {return 'RENDERER_INPUT';}return getParameter(parameter);};"})  Then I thought maybe that it could be something wrong with the parameter number. I configured the command execution without the if statement, but the same thing happened: It worked on my Local IDE but had no effect on an AWS Lambda Server. Simply Put: Is it possible to add Vendor/Renderer on AWS Lambda? In my efforts, it seems that there is no possible way. I made sure to submit this issue on the selenium-stealth GitHub Repository.

Answer: A solution I found for the missing WebGL Vendor/Renderer was using a docker container instead of the normal Lambda layers when creating a function. Not only does the storage increase by a factor of 40X but it also solves the WebGL Vendor/Renderer problem:
Question: I have project where I'm using SQLAlchemy for models and I'm trying to integrate Alembic for making migrations. Everything works as expected when I change models and Alembic sees that models have changed -> it creates good migration file with command: alembic revision --autogenerate -m "model changed" But when I have NOT changed anything in models and I use the same command: alembic revision --autogenerate -m "should be no migration" revision gives me 'empty' revision file like this: """next Revision ID: d06d2a8fed5d Revises: 4461d5328f57 Create Date: 2021-12-02 18:09:42.208607 """ from alembic import op import sqlalchemy as sa # revision identifiers, used by Alembic. revision = 'd06d2a8fed5d' down_revision = '4461d5328f57' branch_labels = None depends_on = None def upgrade(): # ### commands auto generated by Alembic - please adjust! ### pass # ### end Alembic commands ### def downgrade(): # ### commands auto generated by Alembic - please adjust! ### pass # ### end Alembic commands ###  What is purpose of this file? Could I prevent creation of this 'empty file' when alembic revision --autogenerate will not see any changes? To compare when I use Django and it's internal migration when I type command: python manage.py makemigrations I get output something like: No changes detected and there is not migration file created. Is there a way to do the same with Alembic revision? Or is there other command that could check if there were changes in models and if there were then I could simply run alembic revision and upgrade?

Answer: Accepted answer does not answer the question. The correct answer is: Yes, you can call alembic revision --autogenerate and be sure that only if there are changes a revision file would be generated: As per alembic's documentation Implemented in Flask-Migrate (exactly in this file), it's just a change to env.py to account for the needed feature, namely to not autogenerate a revision if there are no changes to the models. You would still run alembic revision --autogenerate -m "should be no migration" but the change you would make to the env.py is, in short:  def run_migrations_online(): # almost identical to Flask-Migrate (Thanks miguel!) # this callback is used to prevent an auto-migration from being generated # when there are no changes to the schema def process_revision_directives(context, revision, directives): if config.cmd_opts.autogenerate: script = directives[0] if script.upgrade_ops.is_empty(): directives[:] = [] print('No changes in schema detected.') connectable = engine_from_config( config.get_section(config.config_ini_section), prefix="sqlalchemy.", poolclass=pool.NullPool, ) with connectable.connect() as connection: context.configure( connection=connection, target_metadata=target_metadata, process_revision_directives=process_revision_directives ) with context.begin_transaction(): context.run_migrations()  Now, you can easily call alembic revision --autogenerate without risking the creation of a new empty revision.
Question: I'm trying to run the deit colab notebook found here: https://colab.research.google.com/github/facebookresearch/deit/blob/colab/notebooks/deit_inference.ipynb but I'm running into an issue in the second cell, specifically the import timm line, which returns this: ImportError: cannot import name 'container_abcs' from 'torch._six'

Answer:  Issue related to this error here:  Try a specific version of timm library: !pip install timm==0.3.2
Question: I have built an API using FastAPI and am trying to send data to it from a client. Both the API and the client use a similar Pydantic model for the data that I want to submit. This includes a field that contains a file path, which I store in a field of type pathlib.path. However, FastAPI does not accept the submission because it apparently cannot handle the path object: TypeError: Object of type PosixPath is not JSON serializable Here's a minimal test file that shows the problem: import pathlib from pydantic import BaseModel from fastapi import FastAPI from fastapi.testclient import TestClient api = FastAPI() client = TestClient(api) class Submission(BaseModel): file_path: pathlib.Path @api.post("/", response_model=Submission) async def add_submission(subm: Submission): print(subm) # add submission to database return subm def test_add_submission(): data = {"file_path": "/my/path/to/file.csv"} print("original data:", data) # create a Submission object, which casts filePath to pathlib.Path: submission = Submission(**data) print("submission object:", submission) payload = submission.dict() print("payload:", payload) response = client.post("/", json=payload) # this throws the error assert response.ok test_add_submission()  When I change the model on the client side to use a string instead of a Path for file_path, things go through. But then I lose the pydantic power of casting the input to a Path when a Submission object is created, and then having a Path attribute with all its possibilities. Surely, there must be better way? What is the correct way to send a pathlib.PosixPath object to a FastAPI API as part of the payload? (This is Python 3.8.9, fastapi 0.68.1, pydantic 1.8.2 on Ubuntu)

Answer: The problem with you code is, that you first transform the pydantic model into a dict, which you then pass to the client, which uses its own json serializer and not the one provided by pydantic. submission.dict() converts any pydantic model into a dict but keeps any other datatype. With client.post("/", json=payload) requests json serializer is used, which cannot handle pathlib.Path. The solution is, not to convert the pydantic model into dict first, but use the json() method of the pydantic model itself and pass it to the client. response = client.post("/", data=submission.json())  Notice that you have to change the parameter json to data.
Question: I wanted to build a website and embed the jupyter notebook functionality of being able to create cells and run python code within it into my website For creating a website I m using Django and I would like to embed either the google collab or jupyter notebook By the way I have researched enough and have been stuck with the StackOverflow links where there no answer about this or the one where they want to use django in jupyter notebook Thanks in advance for any guidance or any reference that you guys can provide.

Answer: You have many options to do that: Note:: I used "jupyter-lab" you can use "jupyter notebook" 1- The first option to redirect to "jupyter notebook" django view.py from django.shortcuts import redirect,HttpResponse import subprocess import time def open_jupiter_notbook(request): b= subprocess.check_output("jupyter-lab list".split()).decode('utf-8') if "9999" not in b: a=subprocess.Popen("jupyter-lab --no-browser --port 9999".split()) start_time = time.time() unreachable_time = 10 while "9999" not in b: timer = time.time() elapsed_time = timer-start_time b= subprocess.check_output("jupyter-lab list".split()).decode('utf-8') if "9999" in b: break if elapsed_time > unreachable_time: return HttpResponse("Unreachable") path = b.split('\n')[1].split('::',1)[0] #You can here add data to your path if you want to open file or anything return redirect(path)  if you want to implement it in template instead of redirect, you can use the following code in Django template: <iframe src="{% url 'open_jupiter_notbook' %}" width= 600px height=200px></iframe>  2- The second option: just use jupyter notebook commands by using this subprocess.check_output("your command".split())
Question: I am using opencv-python==4.5.1.48 and python3.9 docker. I want to save a video in h264 format. Here is my function to save a video: import cv2 def save_video(frames): fps = 30 video_path = '/home/save_test.mp4' fourcc = cv2.VideoWriter_fourcc(*'h264') video_writer = cv2.VideoWriter(video_path, fourcc, fps, (112, 112)) for frame in frames: video_writer.write(frame) video_writer.release()  When I use .mp4 format to save video, I get the following error:  OpenCV: FFMPEG: tag 0x34363268/'h264' is not supported with codec id 27 and format 'mp4 / MP4 (MPEG-4 Part 14)' OpenCV: FFMPEG: fallback to use tag 0x31637661/'avc1' Could not find encoder for codec id 27: Encoder not found  I search and read some solutions but none of them solve my issue. Update: I also install libx264-dev which was recommended in this post and did not work.

Answer: Finally, I found the solution. I can solve my problem in the ubuntu:20.04 docker. The important thing you should notice is that you should install OpenCV via apt-get install python3-opencv not using pip.
Question: Consider a simple pair of generic classes: T = TypeVar("T", str, int) class Base(Generic[T]): def __init__(self, v: T): self.v: T = v @property def value(self) -> T: return self.v class Child(Base[T]): def __init__(self, v: T): super().__init__(v) x = Child(123) reveal_type(x.value)  While using T = TypeVar("T") works as expected. The restricted TypeVar as shown yields the following errors: error: Argument 1 to "__init__" of "Base" has incompatible type "str"; expected "T" error: Argument 1 to "__init__" of "Base" has incompatible type "int"; expected "T" note: Revealed type is 'builtins.int*'  Note that reveal_type still works. Another difference, is that the restricted TypeVar requires the type annotation for self.v assignment whereas the unrestricted does not. In the full use-case, I actually have Callable[[Any], T], but the issue is the same. This is with mypy 0.910 and Python 3.9.7.

Answer: Bounding T to an Union[int,str] should do the job: T = TypeVar("T", bound=Union[str, int]) class Base(Generic[T]): def __init__(self, v: T): self.v: T = v @property def value(self) -> T: return self.v class Child(Base[T]): def __init__(self, v: T): super().__init__(v) x = Child(123) reveal_type(x.value) y = Child('a') reveal_type(y.value)
Question: I am currently using SQLAlchemy ORM to deal with my db operations. Now I have a SQL command which requires ON CONFLICT (id) DO UPDATE. The method on_conflict_do_update() seems to be the correct one to use. But the post here  says the code have to switch to SQLAlchemy core and the high-level ORM functionalities are missing. I am confused by this statement since I think the code like the demo below can achieve what I want while keep the functionalities of SQLAlchemy ORM. class Foo(Base): ... bar = Column(Integer) foo = Foo(bar=1) insert_stmt = insert(Foo).values(bar=foo.bar) do_update_stmt = insert_stmt.on_conflict_do_update( set_=dict( bar=insert_stmt.excluded.bar, ) ) session.execute(do_update_stmt)  I haven't tested it on my project since it will require a huge amount of modification. Can I ask if this is the correct way to deal with ON CONFLICT (id) DO UPDATE with SQLALchemy ORM?

Answer: As noted in the documentation, the constraint= argument is  The name of a unique or exclusion constraint on the table, or the constraint object itself if it has a .name attribute.  so we need to pass the name of the PK constraint to .on_conflict_do_update(). We can get the PK constraint name via the inspection interface: insp = inspect(engine) pk_constraint_name = insp.get_pk_constraint(Foo.__tablename__)["name"] print(pk_constraint_name) # tbl_foo_pkey new_bar = 123 insert_stmt = insert(Foo).values(id=12345, bar=new_bar) do_update_stmt = insert_stmt.on_conflict_do_update( constraint=pk_constraint_name, set_=dict(bar=new_bar) ) with Session(engine) as session, session.begin(): session.execute(do_update_stmt)
Question: I am trying to use Bazel to rebuild Tensorflow with compiler flags, but have been getting errors. In the documentation explaining how to build Tensorflow from the source, they say that Bazelisk will download the correct version for Tensorflow... but when I was getting the errors, I decided to check the Bazel version and it said 4.2.2 (which is the latest version). The tested builds the Bazel version is only 3.7.2 for 2.7 version of Tensorflow (which is what I am using)... Is there a way to use Bazelisk to install version 3.7.2? Or do I have to manually download Bazel and add it to the path?  If so, how do I do it? The way I installed it before was just downloading and running the .exe file from the website... and there wasn't any opportunity to enter a version for Bazel... If not, how do I uninstall Bazelisk? I tried looking for a way to uninstall, but couldn't find anything... 

Answer: Bazelisk will look in the WORKSPACE root for a file named .bazelversion (see here). This file is supposed to contain the Bazel version number you want to use. There are also other options to tell Bazelisk which version to use: How does Bazelisk know which Bazel version to run? To use for instance Bazel 0.26.1 you can Bazelisk this way: $ USE_BAZEL_VERSION=0.26.1 bazelisk version
Question: I'm using fastAPI python framework to build a simple POST/GET https server. On the client side, i send heartbeat POST messages every 10 seconds and i'd like to keep my connection open during this period. However, for some reason I see that every new heartbeat, my connection get disconnected by the peer, so I need to re-establish it. If the idle period between 2 consecutive keepalives is 1 second, the Connection remains active, and can be reused. I'm using HTTP/1.1 with Connection: keep-alive, but it is entirely up to the server how long it will keep the connection alive, and I'm looking for a way to extend this timeout to ~15 seconds. is there any suitable way to do it ? or even just make the server print proper log message when it decide to disconnect the client peer ... P.S. in order to start the server I'm using the following command, Perhaps it need to be modified ? uvicorn main:app --port 44444 --host 0.0.0.0 --reload --ssl-keyfile ./key.pem --ssl-certfile ./certificate.pem --log-level debug 

Answer: From the uvicorn docs: --timeout-keep-alive <int> - Close Keep-Alive connections if no new data is received within this timeout. Default: 5. But it‘s probably no great idea to set this to 10 minutes. What is your problem with dropping the connection for a heartbeat?
Question: I am currently converting a script from Python 2 to Python 3. While debugging it, I stumbled upon a part of the code that behaves differently between both versions. However, I am unable to explain this difference. Here is a reproducer: variable_1 = "x" variable_2 = "y" list_of_variables = ['variable_1', 'variable_2'] existing_variables = vars() print([variable for variable in list_of_variables if variable in vars()]) print([variable for variable in list_of_variables if variable in existing_variables])  Python 2.7.18 shows the following output: ['variable_1', 'variable_2'] ['variable_1', 'variable_2']  Whereas Python 3.9.0 displays: [] ['variable_1', 'variable_2']  Why is the first list comprehension not working in Python 3? And why is it working when storing the content of vars() within a variable?

Answer: It's working in both: it's just working differently. In Python 3, list comprehensions create their own local scope to avoid leaking variable names into the calling scope. The call to vars() inside the list comprehension is just returning the variables defined in the list comprehension's own scope, not the scope where the list comprehension is used. From https://docs.python.org/3/reference/expressions.html#displays-for-lists-sets-and-dictionaries:  However, aside from the iterable expression in the leftmost for clause, the comprehension is executed in a separate implicitly nested scope. This ensures that names assigned to in the target list don’t “leak” into the enclosing scope.
Question: In case of invalid parameters, cv2.VideoWriter writes stuff to stderr. here is a minimal example: import cv2 cv2.VideoWriter("foo.mp4", cv2.VideoWriter_fourcc(*"mp4v"), 90000, (240, 240))  Running like so python3 video_verification/verification.py 2> err.txt  one can see the expected errors in err.txt: [mpeg4 @ 0x13e4640] timebase 1/90000 not supported by MPEG 4 standard, the maximum admitted value for the timebase denominator is 65535 [ERROR:0] global /tmp/pip-req-build-13uokl4r/opencv/modules/videoio/src/cap_ffmpeg_impl.hpp (2705) open Could not open codec mpeg4, error: Unspecified error [ERROR:0] global /tmp/pip-req-build-13uokl4r/opencv/modules/videoio/src/cap_ffmpeg_impl.hpp (2722) open VIDEOIO/FFMPEG: Failed to initialize VideoWriter  However, I'm unable to capture this directly in Python. Here are my two (failing) attemps so far: import io from contextlib import redirect_stdout, redirect_stderr import cv2 f_out = io.StringIO() f_err = io.StringIO() with redirect_stdout(f_out): with redirect_stderr(f_err): fps = 90000 writer = cv2.VideoWriter( "foo.mp4", cv2.VideoWriter_fourcc(*"mp4v"), fps, (240, 240) ) print(f"stdout: {f_out.getvalue()}") print(f"stderr: {f_err.getvalue()}")  import io import sys import cv2 def flush(): sys.stdout.flush() sys.stderr.flush() flush() original_stdout = sys.stdout original_stderr = sys.stderr captured_stdout = io.StringIO() captured_stderr = io.StringIO() sys.stdout = captured_stdout sys.stderr = captured_stderr flush() cv2.VideoWriter("foo.mp4", cv2.VideoWriter_fourcc(*"mp4v"), 90000, (240, 240)) flush() sys.stdout = original_stdout sys.stderr = original_stderr print(f"stdout: {captured_stdout.getvalue()}") print(f"stderr: {captured_stderr.getvalue()}")  I would love to learn what I'm doing wrong. :-)

Answer: I've found the wurlitzer library, which can do exactly that, i.e., capture the streams written to by a C library: import cv2 from wurlitzer import pipes with pipes() as (out, err): cv2.VideoWriter("foo.mp4", cv2.VideoWriter_fourcc(*"mp4v"), 90000, (240, 240)) print(f"stderr: {err.read()}")
Question: Reading official documentation of both libraries: os.path  See also The pathlib module offers high-level path objects.  pathlib  See also For low-level path manipulation on strings, you can also use the os.path module.  So os.path is for low-level use and path lib for high level use. But what it means? I don't understand the low vs high in this context. There are many examples out there about pathlib vs os.path for example Pathlib vs. os.path.join in Python but in none of them there is explanation of the low vs high concept. Is there example that explain what exactly is low and what exactly is high? Is it correct that low level means the need to manipulate the path as string object?

Answer: I see it as "Use pathlib if you are interested in working with the path as an abstract entity" , use "os.path" if you're really interested in the resources represented by that path; in most cases you are interested in the using the 'path' to learn about or handle resources. High level being 'the path as an abstract entity' low level being 'the resources represented by a path'. While os.path also does have functionality for pathname manipulation etc. it's all there to enable you to get to the resources you're really interested in. From the pathlib docs:  If you’ve never used this module before or just aren’t sure which class is right for your task, Path is most likely what you need. It instantiates a concrete path for the platform the code is running on. Pure paths are useful in some special cases; for example: If you want to manipulate Windows paths on a Unix machine (or vice versa). You cannot instantiate a WindowsPath when running on Unix, but you can instantiate PureWindowsPath. You want to make sure that your code only manipulates paths without actually accessing the OS. In this case, instantiating one of the pure classes may be useful since those simply don’t have any OS-accessing operations.
Question: I know Apple provides tensorflow-metal on their conda -c apple channel. How do I see the rest of the packages in that channel? The "answers" I found online address finding a package to install, like conda search *tensorflow* but this is not what I want. Instead, I would like to see what packages Apple has in their -c apple channel that I can install: in short "browse apple repository", a concept that I'm surprised is so difficult to achieve under conda... p.s. I have tried conda search -c apple but it only lists all the packages from conda-forge, even after I did a conda config --add channels apple.

Answer: Web Browser You can browse all packages in an Anaconda Cloud channel by visiting: https://anaconda.org/<channel_name>/repo  For the apple channel, the tensorflow-deps package is the only package (right now). Conda CLI Alternatively, one can also use conda search with something like: $ conda search --override-channels -c apple '*' Loading channels: done # Name Version Build Channel tensorflow-deps 2.5.0 0 apple tensorflow-deps 2.5.0 1 apple tensorflow-deps 2.6.0 0 apple tensorflow-deps 2.7.0 0 apple  to list all packages. Note that the selector * must be escaped ('*') otherwise the shell will interpret it. PyPI The tensorflow-macos and tensorflow-metal packages are on PyPI, not Anaconda Cloud, and are uploaded by user Kulin Seth.