eshangj/stackoverflow_q_and_a_sample huggingface dataset:
Question: I have Glue job, a python shell code. When I try to run it I end up getting the below error. Job Name : xxxxx Job Run Id : yyyyyy failed to execute with exception Internal service error : Invalid input provided It is not specific to code, even if I just put import boto3 print('loaded')  I am getting the error right after clicking the run job option. What is the issue here?

Answer: I think Quatermass is right, the jobs started working out of the blue the next day without any changes.
Question: I have a Numpy array of dimensions (d1,d2,d3,d4), for instance A = np.arange(120).reshape((2,3,4,5)). I would like to contract it so as to obtain B of dimensions (d1,d2,d4). The d3-indices of parts to pick are collected in an indexing array Idx of dimensions (d1,d2). Idx provides, for each couple (x1,x2) of indices along (d1,d2), the index x3 for which B should retain the whole corresponding d4-line in A, for example Idx = rng.integers(4, size=(2,3)). To sum up, for all (x1,x2), I want B[x1,x2,:] = A[x1,x2,Idx[x1,x2],:]. Is there an efficient, vectorized way to do that, without using a loop? I'm aware that this is similar to Easy way to do nd-array contraction using advanced indexing in Python but I have trouble extending the solution to higher dimensional arrays. MWE A = np.arange(120).reshape((2,3,4,5)) Idx = rng.integers(4, size=(2,3)) # correct result: B = np.zeros((2,3,5)) for i in range(2): for j in range(3): B[i,j,:] = A[i,j,Idx[i,j],:] # what I would like, which doesn't work: B = A[:,:,Idx[:,:],:] 

Answer: Times for 3 alternatives: In [91]: %%timeit ...: B = np.zeros((2,3,5),A.dtype) ...: for i in range(2): ...: for j in range(3): ...: B[i,j,:] = A[i,j,Idx[i,j],:] ...: 11 µs ± 48.8 ns per loop (mean ± std. dev. of 7 runs, 100,000 loops each) In [92]: timeit A[np.arange(2)[:,None],np.arange(3),Idx] 8.58 µs ± 44 ns per loop (mean ± std. dev. of 7 runs, 100,000 loops each) In [94]: timeit np.squeeze(np.take_along_axis(A, Idx[:,:,None,None], axis=2), axis=2) 29.4 µs ± 448 ns per loop (mean ± std. dev. of 7 runs, 10,000 loops each)  Relative times may differ with larger arrays. But this is a good size for testing the correctness.
Question: I've seen a few variations of running a query with SQLAlchemy. For example, here is one version: posts = db.query(models.Post).filter(models.Post.owner_id==user.id).all()  What would be the difference between using the above or using .where? Why are there two variations here?

Answer: According to the documentation, there is no difference.  method sqlalchemy.orm.Query.where(*criterion) A synonym for Query.filter().  It was added in version 1.4 by this commit. According to the commit message the reason to add it was to Convert remaining ORM APIs to support 2.0 style. You can read more about "2.0 style" here.
Question: I came across a code snippet for declaring Pydantic Models. The inheritance used there has me confused. class RecipeBase(BaseModel): label: str source: str url: HttpUrl class RecipeCreate(RecipeBase): label: str source: str url: HttpUrl submitter_id: int class RecipeUpdate(RecipeBase): label: str  I am not sure what's the benefit of inheriting from RecipeBase in the RecipeCreate and RecipeUpdate class. The part that has me confused is that after inheritance also, why does one has to re-declare label, source, and URL, which are already part of the RecipeBase class in the RecipeCreate class?

Answer: I’d say it is an oversight from the tutorial. There is no benefit and only causes confusion. Typically, Base is used for all overlapping fields, and they are only overloaded when they change type (for example, XyzBase has name: str whereas XyzCreate has name: str|None because it doesn’t has to be provided when updating an instance. The tutorial is doing a bad job explaining why the setup is as it is.
Question: Here is my code, and i am getting an AttributeError: 'tuple' object has no attribute 'sort. I am trying to do an image alignment and found this standard image alignment code in an article. I am learning openCV and python which i am really new too, I am able to do basic stuff with openCV right now i am trying to learn image alignment and i am stuck on this part. Traceback (most recent call last): File "test9.py", line 31, in <module> matches.sort(key = lambda x: x.distance) AttributeError: 'tuple' object has no attribute 'sort' ------------------ (program exited with code: 1) Press return to continue import cv2 import numpy as np # Open the image files. img1_color = cv2.imread("/home/pi/Desktop/Project AOI/testboard1/image_2.jpg") # Image to be aligned. img2_color = cv2.imread("/home/pi/Desktop/Project AOI/testboard1/image_0.jpg") # Reference image. # Convert to grayscale. img1 = cv2.cvtColor(img1_color, cv2.COLOR_BGR2GRAY) img2 = cv2.cvtColor(img2_color, cv2.COLOR_BGR2GRAY) height, width = img2.shape # Create ORB detector with 5000 features. orb_detector = cv2.ORB_create(5000) # Find keypoints and descriptors. # The first arg is the image, second arg is the mask # (which is not required in this case). kp1, d1 = orb_detector.detectAndCompute(img1, None) kp2, d2 = orb_detector.detectAndCompute(img2, None) # Match features between the two images. # We create a Brute Force matcher with # Hamming distance as measurement mode. matcher = cv2.BFMatcher(cv2.NORM_HAMMING, crossCheck = True) # Match the two sets of descriptors. matches = matcher.match(d1, d2) # Sort matches on the basis of their Hamming distance. matches.sort(key = lambda x: x.distance) # Take the top 90 % matches forward. matches = matches[:int(len(matches)*0.9)] no_of_matches = len(matches) # Define empty matrices of shape no_of_matches * 2. p1 = np.zeros((no_of_matches, 2)) p2 = np.zeros((no_of_matches, 2)) for i in range(len(matches)): p1[i, :] = kp1[matches[i].queryIdx].pt p2[i, :] = kp2[matches[i].trainIdx].pt # Find the homography matrix. homography, mask = cv2.findHomography(p1, p2, cv2.RANSAC) # Use this matrix to transform the # colored image wrt the reference image. transformed_img = cv2.warpPerspective(img1_color, homography, (width, height)) # Save the output. cv2.imwrite('output.jpg', transformed_img) 

Answer: You're getting a tuple returned, not a list. You can't just matches.sort(...) that. OpenCV, since v4.5.4, exhibits this behavior in its Python bindings generation. You have to use this instead: matches = sorted(matches, ...)  This creates a new list, which contains the sorted elements of the original tuple. Related issues:  https://github.com/opencv/opencv/issues/21266 https://github.com/opencv/opencv/issues/21284
Question: Is it possible to get the cookies when someone hits the API? I need to read the cookies for each request. @app.get("/") async def root(text: str, sessionKey: str = Header(None)): print(sessionKey) return {"message": text+" returned"} if __name__ == "__main__": uvicorn.run("main:app", host="0.0.0.0", port=5001 ,reload=True) 

Answer: You can do it in the same way you are accessing the headers in your example (see docs): from fastapi import Cookie @app.get("/") async def root(text: str, sessionKey: str = Header(None), cookie_param: int | None = Cookie(None)): print(cookie_param) return {"message": f"{text} returned"}
Question: I'm reading an article about tuning hyperparameters in keras tuner. It includes code to build a model that has this code: def build_model(hp): """ Builds model and sets up hyperparameter space to search. Parameters ---------- hp : HyperParameter object Configures hyperparameters to tune. Returns ------- model : keras model Compiled model with hyperparameters to tune. """ # Initialize sequential API and start building model. model = keras.Sequential() model.add(keras.layers.Flatten(input_shape=(28,28))) # Tune the number of hidden layers and units in each. # Number of hidden layers: 1 - 5 # Number of Units: 32 - 512 with stepsize of 32 for i in range(1, hp.Int("num_layers", 2, 6)): model.add( keras.layers.Dense( units=hp.Int("units_" + str(i), min_value=32, max_value=512, step=32), activation="relu") ) # Tune dropout layer with values from 0 - 0.3 with stepsize of 0.1. model.add(keras.layers.Dropout(hp.Float("dropout_" + str(i), 0, 0.3, step=0.1))) # Add output layer. model.add(keras.layers.Dense(units=10, activation="softmax")) # Tune learning rate for Adam optimizer with values from 0.01, 0.001, or 0.0001 hp_learning_rate = hp.Choice("learning_rate", values=[1e-2, 1e-3, 1e-4]) # Define optimizer, loss, and metrics model.compile(optimizer=keras.optimizers.Adam(learning_rate=hp_learning_rate), loss=keras.losses.SparseCategoricalCrossentropy(), metrics=["accuracy"]) return model  I'm confused about what the numbers, 1,2 and 6 mean in the range function for the num_layers line.

Answer: If you see the docs, the 2 and 6 are referring to the min and max values respectively. Also note:  [...] max_value is included in the possible values this parameter can take on  So this line: for i in range(1, hp.Int("num_layers", 2, 6)):  basically means: generate a x number of Dense layers, where x is between 1 and 5 and the range from 2 to 6 is the hyperparameter you are working on / tuning.
Question: Take the following code: from sqlalchemy import create_engine from sqlalchemy.orm import declarative_base from sqlalchemy import Column, Integer, String engine = create_engine('postgresql://postgres:password@localhost:5432/db', echo=True, echo_pool='debug') Base = declarative_base() class Item(Base): __tablename__ = 'items' id = Column(Integer, primary_key=True) name = Column(String) def __repr__(self): return "<Item(id=%s, name='%s')>" % (self.id, self.name) item = Item(name="computer")  Is there a way to get a python dict of the Item with all its fields? For example, I would like to get the following returned: item.to_dict() {"id": None, "name": "computer"}  Or do I have to write my own method to do that?

Answer: Here would be one way to do it: class MyBase(Base): __abstract__ = True def to_dict(self): return {field.name:getattr(self, field.name) for field in self.__table__.c} class Item(MyBase): # as before item = Item(name="computer") item.to_dict() # {'id': None, 'name': 'computer'}  Also, a lot of these usability simplifications can be found in: https://github.com/absent1706/sqlalchemy-mixins.
Question: I am trying to reproduce a C++ high precision calculation in full python, but I got a slight difference and I do not understand why. Python: from decimal import * getcontext().prec = 18 r = 0 + (((Decimal(0.95)-Decimal(1.0))**2)+(Decimal(0.00403)-Decimal(0.00063))**2).sqrt() # r = Decimal('0.0501154666744709107')  C++: #include <iostream> #include <math.h> int main() { double zx2 = 0.95; double zx1 = 1.0; double zy2 = 0.00403; double zy1 = 0.00063; double r; r = 0.0 + sqrt((zx2-zx1)*(zx2-zx1)+(zy2-zy1)*(zy2-zy1)); std::cout<<"r = " << r << " ****"; return 0; } // r = 0.050115466674470907 ****  There is this 1 showing up near the end in python but not in c++, why ? Changing the precision in python will not change anything (i already tried) because, the 1 is before the "rounding". Python: 0.0501154666744709107 C++ : 0.050115466674470907  Edit: I though that Decimal would convert anything passed to it into a string in order to "recut" them, but the comment of juanpa.arrivillaga made me doubt about it and after checking the source code, it is not the case ! So I changed to use string. Now the Python result is the same as WolframAlpha shared by Random Davis: link.

Answer: The origin of the discrepancy is that Python Decimal follows the more modern IBM's General Decimal Arithmetic Specification. In C++ however there too exist support available for 80-bit "extended precision" through the long double format. For reference, the standard IEEE-754 floating point doubles contain 53 bits of precision. Here below the C++ example from the question, refactored using long doubles: #include <iostream> #include <math.h> #include <iomanip> int main() { long double zx2 = 0.95; long double zx1 = 1.0; long double zy2 = 0.00403; long double zy1 = 0.00063; long double r; r = 0.0 + sqrt((zx2-zx1)*(zx2-zx1)+(zy2-zy1)*(zy2-zy1)); std::fixed; std::cout<< std::setprecision(25) << "r = " << r << " ****"; //25 floats // prints "r = 0.05011546667447091067728042 ****" return 0; }
Question: I have a FastAPI server that communicates with a web app. My web app also has 2 types of users, Users (non-admins) and Admins. I added a global dependency to FastAPI to verify the user. I want the verify dependency to only allow Admins to access endpoints by default, and have some decorator (or something similar) to allow non-admins to access certain routes. This way, no one accidentally creates a public route that is supposed to be only for admins.  def verify_token(request: Request): # make sure the user's auth token is valid # retrieve the user's details from the database # make sure user is Admin, otherwise throw HTTP exception return True app = FastAPI( title="My App", dependencies=[Depends(verify_token)] ) @app.get(/admins_only) def admins_only(): # this works well! return {'result': 2} @app.get(/non_admin_route) def non_admin_route(): # this doesn't work because verify_token # only allows admins by default, but it should # be accessible to non admins return {'result': 1} 

Answer: You cannot have conditional global dependencies. You either have them on all endpoints of your app, or on none of them. My recommendation is to split your endpoints in two routers, and only add routes to the respective routers. Then you can add a global dependency to only one of the routers like this: from fastapi import APIRouter, FastAPI, Request, Depends def verify_token(request: Request): # make sure the user's auth token is valid # retrieve the user's details from the database # make sure user is Admin, otherwise throw HTTP exception return True app = FastAPI( title="My App", ) only_admin_router = APIRouter( tags=["forAdmins"], dependencies=[Depends(verify_token)] ) all_users_router = APIRouter(tags="forEverybody") @only_admin_router.get("/admins_only") def admins_only(): # this will only work if verify doesn't raise. return {'result': 2} @all_users_router.get("/non_admin_route") def non_admin_route(): #this will work for all users, verify will not be called. return {'result': 1} app.include_router(only_admin_router) app.include_router(all_users_router)
Question: I have a dictionary for which the key "name" is initialized to None (as this be easily used in if name: blocks) if a name is read in it is then assigned to name. All of this works fine but Pycharm throws a warning when "name" is changed due to the change in type. While this isn't the end of the world it's a pain for debugging (and could be a pain for maintaining the code). Does anyone know if there is a way either to provide something akin to a type hint to the dictionary or failing that to tell Pycharm the change of type is intended? code replicating issue: from copy import deepcopy test = { "name": None, "other_variables": "Something" } def read_info(): test_2 = deepcopy(test) test_2["name"] = "this is the name" # Pycharm shows warning return test_2["name"]  ideal solution: from copy import deepcopy test = { "name": None type=str, "other_variables": "Something" } def read_info(): test_2 = deepcopy(test) test_2["name"] = "this is the name" # no warning return test_2["name"]  Note: I know that setting the default value to "" would behave the same but a) it's handy having it print out "None" if name is printed before assignment and b) I find it slightly more readable to have None instead of "". Note_2: I am unaware why (it may be a bug or intended for some reason I don't understand) but Pycharm only gives a wanrning if the code shown above is found within a function. i.e. replacing the read_info() function with the lines: test_2 = deepcopy(test) test_2["name"] = "this is the name" # Pycharm shows warning  Does not give a warning

Answer: Type hinting that dictionary with dict[str, None | str] (Python 3.10+, older versions need to use typing.Dict[str, typing.Optional[str]]) seems to fix this: from copy import deepcopy test: dict[str, None | str] = { "name": None, "other_variables": "Something" } def read_info(): test_2 = deepcopy(test) test_2["name"] = "this is the name" # no warning return test_2["name"]  As noticed by @Tomerikoo simply type hinting as dict also works (this should work on all? Python versions that support type hints too): test: dict = { "name": None, "other_variables": "Something" }
Question: let's say we have the following code example where we create two basic dataframes: import pandas as pd # Creating Dataframes a = [{'Name': 'abc', 'Age': 8, 'Grade': 3}, {'Name': 'xyz', 'Age': 9, 'Grade': 3}] df1 = pd.DataFrame(a) b = [{'ID': 1,'Name': 'abc', 'Age': 8}, {'ID': 2,'Name': 'xyz', 'Age': 9}] df2 = pd.DataFrame(b) # Printing Dataframes display(df1) display(df2)  We get the following datasets:  Name Age Grade 0 abc 8 3 1 xyz 9 3 ID Name Age 0 1 abc 8 1 2 xyz 9  How can I find the list of columns that are not repeated in these frames when they are intersected? That is, as a result, I want to get the names of the following columns: ['Grade', 'ID']

Answer: Use symmetric_difference res = df2.columns.symmetric_difference(df1.columns) print(res)  Output Index(['Grade', 'ID'], dtype='object')  Or as an alternative, use set.symmetric_difference res = set(df2.columns).symmetric_difference(df1.columns) print(res)  Output {'Grade', 'ID'}  A third alternative, suggested by @SashSinha, is to use the shortcut: res = df2.columns ^ df1.columns  but as of pandas 1.4.3 this issue a warning:  FutureWarning: Index.xor operating as a set operation is deprecated, in the future this will be a logical operation matching Series.xor. Use index.symmetric_difference(other) instead. res = df2.columns ^ df1.columns
Question: I've implemented validation_epoch_end to produce and log metrics, and when I run trainer.validate, the metrics appear in my notebook. However, when I run trainer.fit, only the training metrics appear; not the validation ones. The validation step is still being run (because the validation code calls a print statement, which does appear), but the validation metrics don't appear, even though they're logged. Or, if they do appear, the next epoch immediately erases them, so that I can't see them. (Likewise, tensorboard sees the validation metrics) How can I see the validation epoch end metrics in a notebook, as each epoch occurs?

Answer: You could do the following. Let's say you have the following LightningModule: class MNISTModel(LightningModule): def __init__(self): super().__init__() self.l1 = torch.nn.Linear(28 * 28, 10) def forward(self, x): return torch.relu(self.l1(x.view(x.size(0), -1))) def training_step(self, batch, batch_nb): x, y = batch loss = F.cross_entropy(self(x), y) # prog_bar=True will display the value on the progress bar statically for the last complete train epoch self.log("train_loss", loss, on_step=False, on_epoch=True, prog_bar=True) return loss def validation_step(self, batch, batch_nb): x, y = batch loss = F.cross_entropy(self(x), y) # prog_bar=True will display the value on the progress bar statically for the last complete validation epoch self.log("val_loss", loss, on_step=False, on_epoch=True, prog_bar=True) return loss def configure_optimizers(self): return torch.optim.Adam(self.parameters(), lr=0.02)  The trick is to use prog_bar=True in combination with on_step and on_epoch depending on when you want the update on the progress bar. So, in this case, when training: # Train the model ⚡ trainer.fit(mnist_model, MNIST_dm)  you will see: Epoch 4: 100% -------------------------- 939/939 [00:09<00:00, 94.51it/s, loss=0.636, v_num=4, val_loss=0.743, train_loss=0.726]  Where loss will be updating each batch as it is the step loss. However, val_loss and train_loss will be static values that will only change after each validation or train epoch respectively.
Question: When I run poetry show - most of my packages are blue but a few are red? What do these two colors mean? I think red means the package is not @latest ?

Answer:  Black: Not required package Red: Not installed / It needs an immediate semver-compliant upgrade Yellow: It needs an upgrade but has potentially breaking changes so is not urgent Green: Already latest
Question: I'm working with a request of a remote webhook where the data I want to validate is either there, or an empty dictionary. I would like it to run through the model validator if it's there but also not choke if it's an empty dictionary. input 1: { "something": {} }  input 2: { "something": { "name": "George", } }  input 3: { "something": { "invalid": true } }  class Person(BaseModel): name: str class WebhookRequest(BaseModel): something: Union[Person, Literal[{}]] # invalid literal  How would I model something like this in Pydantic such that inputs 1 and 2 succeed while input 3 fails?

Answer: Use extra = "forbid" option to disallow extra fields and use an empty model to represent an empty dictionary. from pydantic import BaseModel, ConfigDict class Empty(BaseModel): ... model_config = ConfigDict(extra="forbid") class Person(BaseModel): name: str model_config = ConfigDict(extra="forbid") class WebhookRequest(BaseModel): something: Person | Empty model_config = ConfigDict(extra="forbid")
Question: I have two data frames that I would like to each write to its own sheet in an Excel file. The following code accomplishes what I want: import pandas as pd df_x = pd.DataFrame({'a':[1, 2, 3]}) df_y = pd.DataFrame({'b':['a', 'b', 'c']}) writer = pd.ExcelWriter('df_comb.xlsx', engine='xlsxwriter') df_x.to_excel(writer, sheet_name='df_x', index=False) df_y.to_excel(writer, sheet_name='df_y', index=False) writer.save() writer.close()  However, in my actual use case, I have a large number of dataframes and do not want to write a to_excel statement for each. Is there anyway to loop over a list of dataframes to accomplish this, something along the lines of: for i in [df_x, df_y]: i.to_excel(writer, sheet_name = i, index=False) 

Answer: What you have is almost there, I think you'll run into problems trying to assign the sheet_name to be the DataFrame as well. I would suggest also having a list of names that you'd like the sheets to be. You could then do something like this: names = ["df_x", "df_y"] dataframes = [df_x, df_y] for i, frame in enumerate(dataframes): frame.to_excel(writer, sheet_name = names[i], index=False)  If you aren't familiar with it, enumerate makes a tuple where the values are the index and the item in the list, allowing you to index another list. Below is a solution using a dictionary of the dataframes and looping across dictionary.items(). dataframes = {"df_x": df_x, "df_y": df_y} for name, frame in dataframes.items(): frame.to_excel(writer, sheet_name = name, index=False)
Question: I'd like to remove an environment (see this question) when I issue /progetti/project_blah$ poetry env remove ./.venv  I get /bin/sh: 1: ./.venv: Permission denied EnvCommandError Command ./.venv -c "import sys; print('.'.join([str(s) for s in sys.version_info[:3]]))" errored with the following return code 126, and output: at ~/.local/share/pypoetry/venv/lib/python3.9/site-packages/poetry/utils/env.py:625 in remove 621│ shell=True, 622│ ) 623│ ) 624│ except CalledProcessError as e: → 625│ raise EnvCommandError(e) 626│ 627│ python_version = Version.parse(python_version.strip()) 628│ minor = "{}.{}".format(python_version.major, python_version.minor) 629│  What is this ?

Answer: Besides the permission error, if you are here because you're unable to delete a locally created environment using poetry, you can refer to this github issue. Basically, at the moment, you can't delete the local environment with the poetry env remove <python> command since it will return the Environment does not exist error. To overcome the issue you can delete the .venv folder as suggested also in the comment to the question or run poetry env remove --all. Be careful with this last command and use poetry env list before to check that there is only one environment (the locally created one) that is going to be deleted. UPDATE: Problem seems to be fixed now. Check your version of poetry before.