eshangj/stackoverflow_q_and_a_sample huggingface dataset:
Question: I am trying to get a composite index working between a polymorphic subclass and it's parent. Alembic autogenerate does not seem to detect Indexes outside of __table_args__. I can't use __table_args__ because, being in the subclass, it does not count my class as having a __table__. How do I create a composite Index between these? class Main(Base, SomeMixin): __tablename__ = "main" __table_args__ = ( # Some constraints and Indexes specific to main ) id = Column(String, primary_key=True, default=func.generate_object_id()) mtype = Column(String, nullable=False) __mapper_args__ = {"polymorphic_on": mtype} class SubClass(Main): __mapper_args__ = {"polymorphic_identity": "subclass"} bid = Column(String, ForeignKey("other.id", ondelete="CASCADE")) # My index specific to Subclass Index( "ix_main_bid_mtype", "bid", "mtype", )  The goal is to have something like this pop with alembic autogenerate: def upgrade(): # ### commands auto generated by Alembic - please adjust! ### op.create_index( "ix_main_bid_mtype", "main", ["bid", "mtype"], unique=False, ) # ### end Alembic commands ### def downgrade(): # ### commands auto generated by Alembic - please adjust! ### op.drop_index(op.f("ix_main_bid_mtype"), table_name="main") # ### end Alembic commands ###  Thank you for your time and potential future help. EDIT: Note: The other fields are detected by autogenerate, only the index done this way does not seem to work.

Answer: Create the index externally after both classes: class Main(Base, SomeMixin): __tablename__ = "main" __table_args__ = ( # Some constraints and Indexes specific to main ) id = Column(String, primary_key=True, default=func.generate_object_id()) mtype = Column(String, nullable=False) __mapper_args__ = {"polymorphic_on": mtype} class SubClass(Main): __mapper_args__ = {"polymorphic_identity": "subclass"} bid = Column(String, ForeignKey("other.id", ondelete="CASCADE")) Index("ix_main_bid_mtype", SubClass.bid, SubClass.mtype)
Question: I have been using python-poetry for over a year now.  After poetry 1.2.0 release, I get such an info warning: Configuration file exists at ~/Library/Application Support/pypoetry, reusing this directory. Consider moving configuration to ~/Library/Preferences/pypoetry, as support for the legacy directory will be removed in an upcoming release.  But in docs, it is still indicated for macOS: ~/Library/Application Support/pypoetry  https://python-poetry.org/docs/configuration/ My question is that if ~/Library/Preferences/pypoetry is the latest decision what should I do for moving configuration to there?  Is just copy-pasting enough? 

Answer: Looks like it is as simple as copy/pasting to the new directory. I got the same error after upgrading to Poetry 1.2. So I created a pypoetry folder in the new Preferences directory, copy/pasted the config.toml to it, and just to be safe, I renamed the original folder to: ~/Library/Application Support/pypoetry_bak After doing this and running poetry -V, the error is gone.
Question: Is there a way to have the same behavior with Paramiko, as when using ssh option: -o PubkeyAcceptedKeyTypes=+ssh-rsa 

Answer: Paramiko uses ssh-rsa by default. No need to enable it. But if you have problems with public keys, it might be because recent versions of Paramiko first try rsa-sha2-*. And some legacy servers choke on that. So you likely rather want to disable the rsa-sha2-*. For that, see: Paramiko authentication fails with "Agreed upon 'rsa-sha2-512' pubkey algorithm" (and "unsupported public key algorithm: rsa-sha2-512" in sshd log)
Question: I am working on some paper replication, but I am having trouble with it. According to the log, it says that RuntimeError: a view of a leaf Variable that requires grad is being used in an in-place operation. However, when I check the line where the error is referring to, it was just a simple property setter inside the class: @pdfvec.setter def pdfvec(self, value): self.param.pdfvec[self.key] = value # where the error message is referring to  Isn't in-place operations are something like += or *= etc.? I don't see why this error message appeared in this line. I am really confused about this message, and I will be glad if any one knows any possible reason this can happen. For additional information, this is the part where the setter function was called: def _update_params(params, pdfvecs): idx = 0 for param in params: totdim = param.stats.numel() shape = param.stats.shape param.pdfvec = pdfvecs[idx: idx + totdim].reshape(shape) # where the setter function was called idx += totdim  I know this can still lack information for solving the problem, but if you know any possiblity why the error message appeared I would be really glad to hear.

Answer: In-place operation means the assignment you've done is modifiying the underlying storage of your Tensor, of which requires_grad is set to True, according to your error message. That said, your param.pdfvec[self.key] is not a leaf Tensor, because they will be updated during back-propagation. And you tried to assign a value to it , that will interference with autograd, so this action is prohibited by default. You can do this by directly modifying its underlying storage(f.e., with .data).
Question: For example take this app.py and run it with python3 -m flask run --cert=adhoc. from flask import Flask app = Flask(__name__) @app.route('/', methods=["GET", "POST"]) def hello_world(): return {"access_token": "aa" * 50000}  How could sending data truncate the response? >>> import requests >>> len(requests.post('https://127.0.0.1:5000/', verify=False).content) 100020 >>> len(requests.post('https://127.0.0.1:5000/', data={'a':'b'}, verify=False).content) 81920  PS 1: It works as expected without SSL; PS 2: GET produces the same behaviour; PS 3: Curl produces the correct result: $ curl -s -k -X POST https://127.0.0.1:5000/ -H 'Content-Type: application/json' -d '{"a":"b"}' | wc -m 100020  PS 4: I reported this as a bug on requests' github.

Answer: This is a weird one, and I confess I am not entirely sure if my answer is really correct in every detail. But it seems you might need to report this to Flask rather than requests... @app.route('/', methods=["GET", "POST"]) def hello_world(): print(f"hello_world: content_type: {request.content_type}, data: {request.data}") return {"access_token": "aa" * 50000}  This code responds as expected to both >>> len(requests.post('https://127.0.0.1:5000/', data={"a":"b"}, verify=False).content)  and >>> len(requests.post('https://127.0.0.1:5000/', verify=False).content)  Now comment out the print(), and the data=... request gets an exception: raise ChunkedEncodingError(e) requests.exceptions.ChunkedEncodingError: ("Connection broken: ConnectionResetError(104, 'Connection reset by peer')", ConnectionResetError(104, 'Connection reset by peer'))  I suspect that Flask will read the data from the request only on demand. But with SSL, reading data will affect encryption (e.g. via cipher block chaining). So sending a response before fully reading the request leads to encoding problems. So read all the data you get from the request before you send an answer.
Question: I have a csv file with roughly 50K rows of search engine queries. Some of the search queries are the same, just in a different word order, for example "query A this is " and "this is query A". I've tested using fuzzywuzzy's token_sort_ratio function to find matching word order queries, which works well, however I'm struggling with the runtime of the nested loop, and looking for optimisation tips. Currently the nested for loops take around 60 hours to run on my machine. Does anyone know how I might speed this up? Code below: from fuzzywuzzy import fuzz from fuzzywuzzy import process import pandas as pd from tqdm import tqdm filePath = '/content/queries.csv' df = pd.read_csv(filePath) table1 = df['keyword'].to_list() table2 = df['keyword'].to_list() data = [] for kw_t1 in tqdm(table1): for kw_t2 in table2: score = fuzz.token_sort_ratio(kw_t1,kw_t2) if score == 100 and kw_t1 != kw_t2: data +=[[kw_t1, kw_t2, score]] data_df = pd.DataFrame(data, columns=['query', 'queryComparison', 'score'])  Any advice would be appreciated. Thanks!

Answer: Since what you are looking for are strings consisting of identical words (just not necessarily in the same order), there is no need to use fuzzy matching at all. You can instead use collections.Counter to create a frequency dict for each string, with which you can map the strings under a dict of lists keyed by their frequency dicts. You can then output sub-lists in the dicts whose lengths are greater than 1. Since dicts are not hashable, you can make them keys of a dict by converting them to frozensets of tuples of key-value pairs first. This improves the time complexity from O(n ^ 2) of your code to O(n) while also avoiding overhead of performing fuzzy matching. from collections import Counter matches = {} for query in df['keyword']: matches.setdefault(frozenset(Counter(query.split()).items()), []).append(query) data = [match for match in matches.values() if len(match) > 1]  Demo: https://replit.com/@blhsing/WiseAfraidBrackets
Question: Consider the following image:  and the following bounding contour( which is a smooth version of the output of a text-detection neural network of the above image ), so this contour is a given.  I need to warp both images so that I end up with a straight enough textline, so that it can be fed to a text recognition neural network:  using Piecewise Afﬁne Transformation, or some other method. with an implementation if possible or key points of implementation in python. I know how to find the medial axis, order its points, simplify it (e.g using Douglas-Peucker algorithm), and find the corresponding points on a straight line. EDIT: the question can be rephrased -naively- as the following : have you tried the "puppet warp" feature in Adobe Photoshop? you specify "joint" points on an image , and you move these points to the desired place to perform the image warping, we can calculate the source points using a simplified medial axis (e.g 20 points instead of 200 points), and calculate the corresponding target points on a straight line, how to perform Piecewise Afﬁne Transformation using these two sets of points( source and target)? EDIT: modified the images, my bad Papers Here's a paper that does the needed result: A Novel Technique for Unwarping Curved Handwritten Texts Using Mathematical Morphology and Piecewise Affine Transformation another paper: A novel method for straightening curved text-lines in stylistic documents  Similar questions:  Straighten B-Spline Challenge : Curved text extraction using python How to convert curves in images to lines in Python? Deforming an image so that curved lines become straight lines Straightening a curved contour 

Answer: If the goal is to just unshift each column, then: import numpy as np from PIL import Image source_img = Image.open("73614379-input-v2.png") contour_img = Image.open("73614379-map-v3.png").convert("L") assert source_img.size == contour_img.size contour_arr = np.array(contour_img) != 0 # convert to boolean array col_offsets = np.argmax( contour_arr, axis=0 ) # find the first non-zero row for each column assert len(col_offsets) == source_img.size[0] # sanity check min_nonzero_col_offset = np.min( col_offsets[col_offsets > 0] ) # find the minimum non-zero row target_img = Image.new("RGB", source_img.size, (255, 255, 255)) for x, col_offset in enumerate(col_offsets): offset = col_offset - min_nonzero_col_offset if col_offset > 0 else 0 target_img.paste( source_img.crop((x, offset, x + 1, source_img.size[1])), (x, 0) ) target_img.save("unshifted3.png")  with the new input and the new contour from OP outputs this image:
Question: When enabling mypy in vscode ("python.linting.mypyEnabled": true,), then any manual mypy commands become very slow and CPU intensive (10s before vs 3min after). It seems like the two mypy processes should be independent, or even aid each other through a cache, but they seem to be in each other's way. I noticed that from a clean venv it doesn't happen for a while. Only after the vscode has run mypy do the manual mypy commands become slow, even when vscode is not running anymore. The only related question I could find was this.

Answer: I found that making vscode use a different cache directory resolves the problem. Consider adding e.g. the following to your settings.json: "python.linting.mypyArgs": [ "--cache-dir=.mypy_cache/.vscode" ],  Bonus: by remaining within the default directory (mypy_cache), the second cache directory is ignored by git.
Question: Got AttributeError when attempting to get a value for field Firstname in serializer NameSerializer. The serializer field might be named incorrectly and not match any attribute or key on the QuerySet instance. The original exception text was: 'QuerySet' object has no attribute Firstname.  Error:  serializers.py from rest_framework import serializers from .models import Name, ForeName class NameSerializer(serializers.ModelSerializer): class Meta: model = Name fields = '__all__' class ForeNameSerializer(serializers.ModelSerializer): forenames = NameSerializer(many=True, read_only=True) class Meta: model = ForeName fields= '__all__'  models.py from django.db import models import uuid # create your models here class ForeName(models.Model): id = models.UUIDField(primary_key=True, default=uuid.uuid4, editable=False) Forename = models.CharField(max_length=30) def __str__(self): return self.Forename class Name(models.Model): id = models.UUIDField(primary_key=True, default=uuid.uuid4, editable=False) Firstname = models.ForeignKey(ForeName, on_delete=models.PROTECT, related_name="forenames")  views.py from rest_framework.decorators import api_view from rest_framework.response import Response from .serializers import NameSerializer from .models import Name # Create your views here. @api_view(['GET']) def names_list(request): names = Name.objects.all() myname = NameSerializer(names) return Response({"restult": { "Forename" : myname.data, } 

Answer: You need to add many=True in your serializer when initializing with multiple instances. myname = NameSerializer(names,many=True)
Question: I would like to use a permutation-based alternative to scipy.stats.ttest_1samp to test if the mean of my observations is significantly greater than zero. I stumbled upon scipy.stats.permutation_test but I am not sure if this can also be used in my case? I also stumbled upon mne.stats.permutation_t_test which seems to do what I want, but I would like to stick to scipy if I can. Example: import numpy as np from scipy import stats # create data np.random.seed(42) rvs = np.random.normal(loc=5,scale=5,size=100) # compute one-sample t-test t,p = stats.ttest_1samp(rvs,popmean=0,alternative='greater') 

Answer: This test can be performed with permutation_test. With permutation_type='samples', it "permutes" the signs of the observations. Assuming data has been generated as above, the test can be performed as from scipy import stats def t_statistic(x, axis=-1): return stats.ttest_1samp(x, popmean=0, axis=axis).statistic res = stats.permutation_test((rvs,), t_statistic, permutation_type='samples') print(res.pvalue)  If you only care about the p-value, you can get the same result with np.mean as the statistic instead of t_statistic. Admittedly, this behavior for permutation_type='samples' with only one sample is a bit buried in the documentation.  Accordingly, if data contains only one sample, then the null distribution is formed by independently changing the sign of each observation.  But a test producing the same p-value could also be performed as a two-sample test in which the second sample is the negative of the data. To avoid special cases, that's actually what permutation_test does under the hood. In this case, the example code above is a lot faster than permutation_test right now. I'll try to improve that for SciPy 1.10, though.
Question: When I attempt to install a package from our Azure DevOps Artifacts feed, I get the error: Command: pip install org-test-framework --index-url https://<company_url>/tfs/<orgname>/_packaging/<feedname>/pypi/simple/  The keyring prompts for username and password for the site once entered getting the error as below Error: WARNING: 401 Error, Credentials not correct for https://<company_url>/tfs/<org name>/_packaging/<feedname>/pypi/simple/org-test-framework/ ERROR: Could not find a version that satisfies the requirement org-test-framework (from versions: none) ERROR: No matching distribution found for org-test-framework  Note: Same user and password works when I try in the browser and able to download the package directly from above url, so the issue is not my credentials. Also I have tried with everyone in the team and same issue exist!

Answer: You need to use PAT for authenticate as password. Create a Personal access token with Packaging > Read scope to authenticate into Azure DevOps. Refer to this official link for details: https://learn.microsoft.com/en-us/azure/devops/artifacts/quickstarts/python-packages?view=azure-devops#manually-configure-authentication
Question: I'd like to execute code inside my unit tests conditioned on whether they're running from within VSCode or the command line. Is there a way to do so? The reasoning is to add additional visual feedback through cv2.imwrite statements, but to omit these when running a full regression from the command line or when running my CI. I known that I can set a debugging profile inside launch.json and define environment variables there, but this applies only when debugging a unit test:  { "name": "Debug Tests", "type": "python", "request": "test", "console": "integratedTerminal", "python": "${command:python.interpreterPath}", "justMyCode": false, "env": { "MY_ENVIRONMENT_SWITCH_FOR_WRITING_JPEGS": "1" } },  Is there a way to achieve something similar when not running through the debugger?

Answer: Try defining environment variables by using .env files  .env: MY_ENVIRONMENT_SWITCH_FOR_WRITING_JPEGS = 1  test1.py: import os from pathlib import Path from dotenv import find_dotenv, load_dotenv env_path = Path(".") / ".env" load_dotenv(dotenv_path=env_path, verbose=True) print(os.getenv("MY_ENVIRONMENT_SWITCH_FOR_WRITING_JPEGS"))
Question: a = torch.nn.Parameter(torch.ones(5, 5)) a = a.cuda() print(a.requires_grad) b = a b = b - 2 print('a ', a) print('b ', b) loss = (b - 1).pow(2).sum() loss.backward() print(a.grad) print(b.grad)  After executing codes, the a.grad is None although a.requires_grad is True. But if the code a = a.cuda() is removed, a.grad is available after the loss backward.

Answer:  The .grad attribute of a Tensor that is not a leaf Tensor is being accessed. Its .grad attribute won't be populated during autograd.backward(). If you indeed want the gradient for a non-leaf Tensor, use .retain_grad() on the non-leaf Tensor. If you access the non-leaf Tensor by mistake, make sure you access the leaf Tensor instead. See github.com/pytorch/pytorch/pull/30531 for more information.  a = torch.nn.Parameter(torch.ones(5, 5)) a = a.cuda() print(a.requires_grad) b = a b = b - 2 print('a ', a) print('b ', b) loss = (b - 1).pow(2).sum() a.retain_grad() # added this line loss.backward() print(a.grad)  That happens because of your line a = a.cuda() that override the original value of a. You could use a = torch.nn.Parameter(torch.ones(5, 5)) a.cuda()  Or a = torch.nn.Parameter(torch.ones(5, 5, device='cuda'))  a = torch.nn.Parameter(torch.ones(5, 5).cuda())  Or explicitly requesting to retain the gradients of a a.retain_grad() # added this line  Erasing the gradients of intermediate variables can save significant amount of memory. So it is good that you retain gradients only where you need.
Question: Let's suppose that I have a project with this file structure: project_dir └── src ├── package1 │ └── module1.py └── package2 └── module2.py  When I want to use some class from module1 in some other module I type something like class SomeNewClass(ClassFromModule1): # here I press ctrl-space to make auto-import ...  When Pylance makes auto-import it always takes it starting from the root directory of the project, so I got this auto-import: from src.package1.module1 import ClassFromModule1  The problem is that my working directory is src and I want to get from package1.module1 import ClassFromModule1  Is there any way to fix root path of the auto-import Pylance feature?

Answer: By default, if you import a module in src, it should look like the following (same effect as you want).   So please check if the following settings in vscode are modified: Open Settings and search for Analysis: Auto Search Paths.  This item is enabled (checked) by default. If it was modified to be off, turn it back on. Here are some examples: Folder structure: Project_dir ├─ folderone │ └─ package3 │ └─ moudle3.py # --class C ├─ src │ ├─ package1 │ │ └─ moudle1.py # --class A │ └─ package2 │ └─ moudle2.py # --class B └─ Test.py   If you import the modules in the src folder (the above settings are enabled by default): # import code will not include `src` (like the picture shown above) from package1.moudle1 import A   If the above features are turned off, the import code is as follows:   If importing modules under other folders: The import code will start from the first level folder under the current workspace.     Note: vscode uses the currently open folder as the workspace. So which level of folder you open will also affect the code in the import part. The above example is to open the Project_dir folder as a workspace in vscode. If I put Project_dir in a new folder ImpDemo and open the ImpDemo folder in vscode. Then the code of the import part is like this:
Question: I have a type for a dictionary of variables passed to a template: VariablesDict = Dict[str, Union[int, float, str, None]]  Basically, any dictionary where the keys are strings and the values are strings, numbers or None. I use this type in several template related functions. Take this example function: def render_template(name: str, variables: VariablesDict): ...  Calling this function with a dictionary literal works fine: render_template("foo", {"key": "value"})  However, if I assign the dictionary to a variable first, like this: variables = {"key": "value"} render_template("foo", variables)  Mypy gives an error:  Argument 2 to "render_template" has incompatible type "Dict[str, str]"; expected "Dict[str, Union[int, float, str, None]]"  It seems to me that any value of type Dict[str, str] should be safe to pass to a function that expects a parameter of type Dict[str, Union[int, float, str, None]]. Why doesn't that work by default? Is there anything I can do to make this work?

Answer: The reason it doesn't work is that Dict is mutable, and a function which accepts a Dict[str, int|float|str|None] could therefore reasonably insert any of those types into its argument. If the argument was actually a Dict[str, str], it now contains values that violate its type. (For more on this, google "covariance/contravariance/invariance" and "Liskov Substitution Principle" -- as a general rule, mutable containers are invariant over their generic type[s].) As long as render_template doesn't need to modify the dict you pass to it, an easy fix is to have it take a Mapping (which is an abstract supertype of dict that doesn't imply mutability, and is therefore covariant) instead of a Dict: def render_template(name: str, variables: Mapping[str, Union[int, float, str, None]]): ...