eshangj/stackoverflow_q_and_a_sample huggingface dataset:
Question: I have created a package in python, and now I would like to install it as a regular package. What is the difference between just using pip3 install . and pip3 install -e . ?  The reason why I asked, is because with pip3 install . the package, although installed was not seen by the system. While in the second way it was working fine

Answer: The -e flag tells pip to install in editable mode: -e,--editable <path/url> Install a project in editable mode (i.e. setuptools "develop mode") from a local project path or a VCS url.  https://manpages.debian.org/stretch/python-pip/pip.1 So what is editable mode or setuptools "develop mode" ?  This command allows you to deploy your project’s source for use in one or more “staging areas” where it will be available for importing. This deployment is done in such a way that changes to the project source are immediately available in the staging area(s), without needing to run a build or install step after each change. The develop command works by creating an .egg-link file (named for the project) in the given staging area. If the staging area is Python’s site-packages directory, it also updates an easy-install.pth file so that the project is on sys.path by default for all programs run using that Python installation. The develop command also installs wrapper scripts in the staging area (or a separate directory, as specified) that will ensure the project’s dependencies are available on sys.path before running the project’s source scripts. And, it ensures that any missing project dependencies are available in the staging area, by downloading and installing them if necessary. Last, but not least, the develop command invokes the build_ext -i command to ensure any C extensions in the project have been built and are up-to-date, and the egg_info command to ensure the project’s metadata is updated (so that the runtime and wrappers know what the project’s dependencies are). If you make any changes to the project’s setup script or C extensions, you should rerun the develop command against all relevant staging areas to keep the project’s scripts, metadata and extensions up-to-date.  or, tldr;  Deploy your project in “development mode”, such that it’s available on sys.path, yet can still be edited directly from its source checkout.  https://setuptools.readthedocs.io/en/latest/setuptools.html#develop-deploy-the-project-source-in-development-mode
Question: How can I get the second minimum value from each column? I have this array: A = [[72 76 44 62 81 31] [54 36 82 71 40 45] [63 59 84 36 34 51] [58 53 59 22 77 64] [35 77 60 76 57 44]]  I wish to have output like: A = [54 53 59 36 40 44] 

Answer: Try this, in just one line: [sorted(i)[1] for i in zip(*A)]  in action: In [12]: A = [[72, 76, 44, 62, 81, 31], ...: [54 ,36 ,82 ,71 ,40, 45], ...: [63 ,59, 84, 36, 34 ,51], ...: [58, 53, 59, 22, 77 ,64], ...: [35 ,77, 60, 76, 57, 44]] In [18]: [sorted(i)[1] for i in zip(*A)] Out[18]: [54, 53, 59, 36, 40, 44]   zip(*A) will transpose your list of list so the columns become rows.  and if you have duplicate value, for example: In [19]: A = [[72, 76, 44, 62, 81, 31], ...: [54 ,36 ,82 ,71 ,40, 45], ...: [63 ,59, 84, 36, 34 ,51], ...: [35, 53, 59, 22, 77 ,64], # 35 ...: [35 ,77, 50, 76, 57, 44],] # 35  If you need to skip both 35s, you can use set(): In [29]: [sorted(list(set(i)))[1] for i in zip(*A)] Out[29]: [54, 53, 50, 36, 40, 44]
Question: When creating a new Python project, why would I want to select this option? If I don't select it, what functionality am I missing out on? Would I not be able to import certain Python modules?

Answer: It's just an option to pre-install some packages that you're using everytime, or if it doesn't bother you to have extra packages in your local python interpreted  select it : all packages installed in the global python of your machine will be accessible for the interpreter you're going to create in the virtualenv. do not select it : the interpreter you're going to create in the virtualenv will just have the basic, like pip, and setuptools, then you can install just what you need   Python global and venv :  The global python, is the one in /usr/bin in Linux, or wherever in Windows, this is the main installation of the program, and you can add extra packages using pip  When you're working on something, you may need only some packages, or specific version so not using the global Python. You can create a virtualenv, or pyenv, that will link a local python to the global one, for the main python functionnality, but the packages will be installed only in the virtualenv (and when using Pycharm, it can install for you the main package into the virtualenv you're creating)
Question: I'm still working on my understanding of the PyTorch autograd system. One thing I'm struggling at is to understand why .clamp(min=0) and nn.functional.relu() seem to have different backward passes. It's especially confusing as .clamp is used equivalently to relu in PyTorch tutorials, such as https://pytorch.org/tutorials/beginner/pytorch_with_examples.html#pytorch-nn. I found this when analysing the gradients of a simple fully connected net with one hidden layer and a relu activation (linear in the outputlayer). to my understanding the output of the following code should be just zeros. I hope someone can show me what I am missing. import torch dtype = torch.float x = torch.tensor([[3,2,1], [1,0,2], [4,1,2], [0,0,1]], dtype=dtype) y = torch.ones(4,4) w1_a = torch.tensor([[1,2], [0,1], [4,0]], dtype=dtype, requires_grad=True) w1_b = w1_a.clone().detach() w1_b.requires_grad = True w2_a = torch.tensor([[-1, 1], [-2, 3]], dtype=dtype, requires_grad=True) w2_b = w2_a.clone().detach() w2_b.requires_grad = True y_hat_a = torch.nn.functional.relu(x.mm(w1_a)).mm(w2_a) y_a = torch.ones_like(y_hat_a) y_hat_b = x.mm(w1_b).clamp(min=0).mm(w2_b) y_b = torch.ones_like(y_hat_b) loss_a = (y_hat_a - y_a).pow(2).sum() loss_b = (y_hat_b - y_b).pow(2).sum() loss_a.backward() loss_b.backward() print(w1_a.grad - w1_b.grad) print(w2_a.grad - w2_b.grad) # OUT: # tensor([[ 0., 0.], # [ 0., 0.], # [ 0., -38.]]) # tensor([[0., 0.], # [0., 0.]]) # 

Answer: The reason is that relu and clamp produce different gradients at 0. For a scalar tensor x = 0:  (relu(x) - 1.0).pow(2).backward() gives x.grad == 0 (x.clamp(min=0) - 1.0).pow(2).backward() gives x.grad == -2  This indicates that:  relu chooses x == 0 --> grad = 0 clamp chooses x == 0 --> grad = 1
Question: I have a generic class Graph(Generic[T]). Is there a function that returns the type arguments passed to the class Graph? >>> g = Graph[int]() >>> magic_func(g) <class 'int'> 

Answer: Here is one way to achieve this which works from Python 3.6+ (tested it in 3.6, 3.7 and 3.8): from typing import TypeVar, Generic T = TypeVar('T') class Graph(Generic[T], object): def get_generic_type(self): print(self.__orig_class__.__args__[0]) if __name__=='__main__': g_int = Graph[int]() g_str = Graph[str]() g_int.get_generic_type() g_str.get_generic_type()  Output: <class 'int'> <class 'str'>  If you want to get the type inside __new__ or __init__ things get a little bit tricky, see the following post for more info: Generic[T] base class - how to get type of T from within instance? Edit The library pytypes seems to provide a method that allow to get __orig_class__ even from __init__, check the method get_orig_class available here: https://github.com/Stewori/pytypes/blob/master/pytypes/type_util.py
Question: Poetry has a very good version solver, too good sometimes :) I'm trying to use poetry in a project that uses two incompatible packages. However they are incompatible only by declaration as one of them is no longer developed, but otherwise they work together just fine. With pip I'm able to install these in one environment (with an error printed) and it works. Poetry will declare that the dependencies versions can't be resolved and refuse to install anything.  Is there a way to force poetry to install these incompatible dependencies? Thank you!

Answer: No.  Alternative solutions might be:  contacting the offending package's maintainers and asking for a fix + release forking the package and releasing a fix yourself vendoring the package in your source code - there is no need to install it if it's already there, and many of the usual downsides to vendoring disappear if the project in question is not maintained anymore installing the package by hand post poetry install with an installer that has the option to ignore the dependency resolver, like pip (similar to what you're already doing)
Question: We have 1 table with a large amount of data and DBA's partitioned it based on a particular parameter. This means I ended up with Employee_TX, Employee_NY kind of table names. Earlier the models.py was simple as in -- class Employee(Base): __tablename__ = 'Employee' name = Column... state = Column...  Now, I don't want to create 50 new classes for the newly partitioned tables as anyways my columns are the same. Is there a pattern where I can create a single class and then use it in query dynamically? session.query(<Tablename>).filter().all() Maybe some kind of Factory pattern or something is what I'm looking for. So far I've tried by running a loop as  for state in ['CA', 'TX', 'NY']: class Employee(Base): __qualname__ = __tablename__ = 'Employee_{}'.format(state) name = Column... state = Column...  but this doesn't work and I get a warning as - SAWarning: This declarative base already contains a class with the same class name and module name as app_models.employee, and will be replaced in the string-lookup table. Also it can't find the generated class when I do from app_models import Employee_TX This is a flask app with PostgreSQL as a backend and sqlalchemy is used as an ORM

Answer: Got it by creating a custom function like - def get_model(state): DynamicBase = declarative_base(class_registry=dict()) class MyModel(DynamicBase): __tablename__ = 'Employee_{}'.format(state) name = Column... state = Column... return MyModel  And then from my services.py, I just call with get_model(TX)
Question: Running the following poetry shell  returns the following error /home/harshagoli/.poetry/lib/poetry/_vendor/py2.7/subprocess32.py:149: RuntimeWarning: The _posixsubprocess module is not being used. Child process reliability may suffer if your program uses threads. "program uses threads.", RuntimeWarning) The currently activated Python version 2.7.17 is not supported by the project (^3.7). Trying to find and use a compatible version. Using python3 (3.7.5) Virtual environment already activated: /home/harshagoli/.cache/pypoetry/virtualenvs/my-project-0wt3KWFj-py3.7  How can I get past this error? Why doesn't this command work?

Answer: poetry shell is a really buggy command, and this is often talked about among the maintainers. A workaround for this specific issue is to activate the shell manually. It might be worth aliasing the following source $(poetry env info --path)/bin/activate  so you need to paste this into your .bash_aliases or .bashrc alias acpoet="source $(poetry env info --path)/bin/activate"  Now you can run acpoet to activate your poetry env (don't forget to source your file to enable the command)
Question: Flask-Login recommends having an is_safe_url() function after user login: Here is a link to the part of the documentation that discusses this: https://flask-login.readthedocs.io/en/latest/#login-example They link to this snippet but I don't understand how it implements is_safe_url(): https://palletsprojects.com/p/flask/ next = request.args.get('next') if not is_safe_url(next): return abort(400)  This doesn't seem to come with Flask. I'm relatively new to coding. I want to understand:  What exactly is happening when request gets the next argument? What does the is_safe_url() function do to ensure the URL is safe? Does the next URL need to be checked on login only? Or are there other places and times when it is important to include this security measure? And most importantly: is there a reliable is_safe_url() function that I can use with Flask?  Edit: Added link to Flask-Login documentation and included snippet.

Answer: As mentioned in the comments, Flask-Login today had a dead link in the documentation (issue on GitHub). Please note the warning in the original flask snippets documentation:  Snippets are unofficial and unmaintained. No Flask maintainer has curated or checked the snippets for security, correctness, or design.  The snippet is from urllib.parse import urlparse, urljoin def is_safe_url(target): ref_url = urlparse(request.host_url) test_url = urlparse(urljoin(request.host_url, target)) return test_url.scheme in ('http', 'https') and \ ref_url.netloc == test_url.netloc  Now to address your questions:  What exactly is happening when request gets the next argument?  Part of the code we are focusing on here is next = request.args.get('next') return redirect(next or url_for('dashboard'))  which redirects user to dashboard (e.g. after successful login) by default. However, if user tried to reach for e.g. endpoint profile and wasn't logged in we would want to redirect him to the login page. After logging in default redirect would redirect user to dashboard and not to profile where he intended to go. To provide better user experience we can redirect user to his profile page by building URL /login?next=profile, which enables flask to redirect to profile instead of the default dashboard. Since user can abuse URLs we want to check if URL is safe, or abort otherwise.  What does the is_safe_url() function do to ensure the URL is safe?  The snippet in question is a function that ensures that a redirect target will lead to the same server.  Does the next URL need to be checked on login only? Or are there other places and times when it is important to include this security measure?  No, you should check all dangerous URLs. Example of safe URL redirect would be redirect(url_for('index')), since its hardcoded in your program. See examples of safe and dangerous URLs on Unvalidated Redirects and Forwards - OWASP cheatsheet.  And most importantly: is there a reliable is_safe_url() function that I can use with Flask?  There is Django's is_safe_url() bundled as a standalone package on pypi.
Question: I am using python to do some basic image processing, and want to extend it to process a video frame by frame. I get the video as a blob from a server - .webm encoded - and have it in python as a byte string (b'\x1aE\xdf\xa3\xa3B\x86\x81\x01B\xf7\x81\x01B\xf2\x81\x04B\xf3\x81\x08B\x82\x88matroskaB\x87\x81\x04B\x85\x81\x02\x18S\x80g\x01\xff\xff\xff\xff\xff\xff\xff\x15I\xa9f\x99*\xd7\xb1\x83\x0fB@M\x80\x86ChromeWA\x86Chrome\x16T\xaek\xad\xae\xab\xd7\x81\x01s\xc5\x87\x04\xe8\xfc\x16\t^\x8c\x83\x81\x01\x86\x8fV_MPEG4/ISO/AVC\xe0\x88\xb0\x82\x02\x80\xba\x82\x01\xe0\x1fC\xb6u\x01\xff\xff\xff\xff\xff\xff ...).  I know that there is cv.VideoCapture, which can do almost what I need. The problem is that I would have to first write the file to disk, and then load it again. It seems much cleaner to wrap the string, e.g., into an IOStream, and feed it to some function that does the decoding. Is there a clean way to do this in python, or is writing to disk and loading it again the way to go?

Answer: Two years after Rotem wrote his answer there is now a cleaner / easier way to do this using ImageIO. Note: Assuming ffmpeg is in your path, you can generate a test video to try this example using: ffmpeg -f lavfi -i testsrc=duration=10:size=1280x720:rate=30 testsrc.webm import imageio.v3 as iio from pathlib import Path webm_bytes = Path("testsrc.webm").read_bytes() # read all frames from the bytes string frames = iio.imread(webm_bytes, index=None, format_hint=".webm") frames.shape # Output: # (300, 720, 1280, 3) for frame in iio.imiter(webm_bytes, format_hint=".webm"): print(frame.shape) # Output: # (720, 1280, 3) # (720, 1280, 3) # (720, 1280, 3) # ...  To use this you'll need the ffmpeg backend (which implements a solution similar to what Rotem proposed): pip install imageio[ffmpeg]  In response to Rotem's comment a bit of explanation: The above snippet uses imageio==2.16.0. The v3 API is an upcoming user-facing API that streamlines reading and writing. The API is available since imageio==2.10.0, however, you will have to use import imageio as iio and use iio.v3.imiter and iio.v3.imread on versions older than 2.16.0. The ability to read video bytes has existed forever (>5 years and counting) but has (as I am just now realizing) never been documented directly ... so I will add a PR for that soon™ :) On older versions (tested on v2.9.0) of ImageIO (v2 API) you can still read video byte strings; however, this is slightly more verbose: import imageio as iio import numpy as np from pathlib import Path webm_bytes = Path("testsrc.webm").read_bytes() # read all frames from the bytes string frames = np.stack(iio.mimread(webm_bytes, format="FFMPEG", memtest=False)) # iterate over frames one by one reader = iio.get_reader(webm_bytes, format="FFMPEG") for frame in reader: print(frame.shape) reader.close()
Question: I have been using numpy's random functionality for a while, by calling methods such as np.random.choice() or np.random.randint() etc. I just now found about the ability to create a default_rng object, or other Generator objects: from numpy.random import default_rng gen = default_rng() random_number = gen.integers(10)  So far I would have always used  np.random.randint(10)  instead, and I am wondering what the difference between both ways is. The only benefit I can think of would be keeping track of multiple seeds, or wanting to use specific PRNGs, but maybe there are also differences for a more generic use-case?

Answer: numpy.random.* functions (including numpy.random.binomial) make use of a global pseudorandom number generator (PRNG) object which is shared across the application. On the other hand, default_rng() is a self-contained Generator object that doesn't rely on global state. If you don't care about reproducible "randomness" in your application, these two approaches are equivalent for the time being. Although NumPy's new RNG policy discourages the use of global state in general, it did not deprecate any numpy.random.* functions in version 1.17, although a future version of NumPy might. Note also that because numpy.random.* functions rely on a global PRNG object that isn't thread-safe, these functions can cause race conditions if your application uses multiple threads. (Generator objects are not thread-safe, either, but there are ways to generate pseudorandom numbers via multithreading, without the need to share PRNG objects across threads.)
Question: I am learning to program with python and I am having issues with importing from a module in a package. I am usingvisual studio code with Python 3.8.2 64 bit. My Project Directory .vscode ├── ecommerce │ ├── __init__.py │ ├── database.py │ ├── products.py │ └── payments │ ├── __init__.py │ ├── authorizenet.py │ └── paypal.py ├── __init__.py └── main.py  in the ecommerce/products.py file I have: #products.py from .database import Database p = Database(3,2)  So that I can import the Database class from the ecommerce/database.py file. But I get error ImportError : Attempted relative import with no known parent package 

Answer: It seems, from Python docs and experimenting, that relative imports (involving ., .. etc) only work if  the importing module has a __name__ other than __main__, and further, the __name__ of the importing module is pkg.module_name, i.e., it has to be imported from above in the directory hierarchy (to have a parent pkg as part of it's __name__.)  OR the importing module is being specified via module syntax that includes a parent pkg as python -m pkg.module, in which case it's __name__ is still __main__, so it is being run as a script, yet relative imports will work. Here __package__ is set and used to find the parent package while __name__ is __main__; more here. [After all that, it appears that __package__ and sys.path are key to determining if/how relative imports work. __name__ indicates script or module(i.e., __main__ or module_name). __package__ indicates where in the package the relative imports occur with respect to, and the top of __package__ needs to be in sys.path.] So, continuing with @AmitTendulkar 's example, if you run this as > python main.py or > python -m main or > python -m ecommerce.products from the project root directory, or enter interactive python from that root directory and import main, or import ecommerce.products the relative imports in products.py will work. But if you > python products.py or > python -m products from within ecommerce directory, or enter interactive python from that ecommerce directory and import products they will fail. It is helpful to add print("In module products __package__, __name__ ==", __package__, __name__)  etc. in each file to debug. UPDATE: How imports work depend on sys.path and __package__, not on __name__. Issued from /home/jj, > python sub/mod.py has a sys.path, __package__ of /home/jj/sub, None -absolute imports of modules in sys.path work, relative imports fail. > python -m sub.mod has sys.path, __package__ of /home/jj, sub -absolute imports of modules in sys.path work, relative imports work relative to sys.path + __package__. It is more helpful to add import sys print("In module products sys.path[0], __package__ ==", sys.path[0], __package__)  etc. in each file to debug.
Question: My goal is to run tensorflow object detection API and followed the steps in the installation. I install the tensorflow object detection API and protobuf. I have also added the path to protobuf. But the following error shoots up: ImportError: cannot import name 'string_int_label_map_pb2'  Installed protobuf : %%bash cd models/research protoc object_detection/protos/*.proto --python_out=.  A block of code containing the error import statements: from object_detection.utils import ops as utils_ops from object_detection.utils import label_map_util from object_detection.utils import visualization_utils as vis_util 

Answer: Install protoc-3.11.4 from https://github.com/google/protobuf/releases and run protoc object_detection/protos/*.proto --python_out=. as mentioned in the installation instructions. And put this file in in object detection/protos
Question: I try to read metrics in this way:  data, info = mlflow.get_run(run_id) print(data[1].metrics) # example of output: {'loss': 0.01}  But it get only last value. It is possible to read manually all steps of a particular metric?

Answer: I ran into this same problem and was able to do get all of the values for the metric by using using mlflow.tracking.MlflowClient().get_metric_history. This will return every value you logged using mlflow.log_metric(key, value). Quick example (untested) import mlflow trackingDir = 'file:///....' registryDir = 'file:///...' runID = 'my run id' metricKey = 'loss' client = mlflow.tracking.MlflowClient( tracking_uri=trackingDir, registry_uri=registryDir, ) metrics = client.get_metric_history(runID, metricKey)  From the docs  get_metric_history(run_id, key)[source] Return a list of metric objects corresponding to all values logged for a given metric. Parameters run_id – Unique identifier for run key – Metric name within the run Returns A list of mlflow.entities.Metric entities if logged, else empty list from mlflow.tracking import MlflowClient def print_metric_info(history): for m in history: print("name: {}".format(m.key)) print("value: {}".format(m.value)) print("step: {}".format(m.step)) print("timestamp: {}".format(m.timestamp)) print("--") # Create a run under the default experiment (whose id is "0"). Since this is low-level # CRUD operation, the method will create a run. To end the run, you'll have # to explicitly end it. client = MlflowClient() experiment_id = "0" run = client.create_run(experiment_id) print("run_id:{}".format(run.info.run_id)) print("--") # Log couple of metrics, update their initial value, and fetch each # logged metrics' history. for k, v in [("m1", 1.5), ("m2", 2.5)]: client.log_metric(run.info.run_id, k, v, step=0) client.log_metric(run.info.run_id, k, v + 1, step=1) print_metric_info(client.get_metric_history(run.info.run_id, k)) client.set_terminated(run.info.run_id)