eshangj/stackoverflow_q_and_a_sample huggingface dataset:
Question: I have python 3.6 in my venv on PyCharm. However, I want to change that to Python 3.8. I have already installed 3.8, so how do I change my venv python version? I am on windows 10. Changing the version on the project intepreter settings seems to run using the new venv not my existing venv with all the packages I have installed. Attempting to add a new intepreter also results in the "OK" button being greyed out, possibly due to the current venv being not empty.

Answer: You need to create a new virtual environment with the interpreter which version is 3.8.  Go to Settings => Project => Python Interpreter    Click on the vertical 3 dots, and click on "Add".    Select Virtualenv Environment => New Environment    Choose as base interpreter the one which version is 3.8 (the one you just installed)    Click on "OK" => "OK"  Once you have set the new interpreter, PyCharm will warn you that you need update some dependencies based on your requirements.txt file or, in this case, Pipfile.lock (I am using pipenv for this project)    That's it!
Question: from geopy.distance import vincenty  I just installed the geopy package 2.0.0, I want to use geopy.distance.vincenty() as this doc says. However, it returns ImportError: cannot import name 'vincenty' from 'geopy.distance'. And if I try from geopy import distance  it becomes AttributeError: module 'geopy.distance' has no attribute 'vincenty'. About two or three months ago I used this on Google Colab, and it was fine. What happened? Could it be the latest version letting go this attribute?

Answer: Yes, it has been removed. Look at the changelog's Breaking Changes section which contains this entry:  Removed geopy.distance.vincenty, use geopy.distance.geodesic instead.
Question: I'm learning about comparison operators, and I was playing around with True and False statements. I ran the following code in the Python shell: not(5>7) == True  As expected, this returned True. However, I then ran the following code: True == not(5>7)  and there was a syntax error. Why was this? If the first line of code is valid syntax, then surely the second line of code should also be valid. Where have I gone wrong? (To give a bit of background, my understanding is that = in Python is only used for variable assignment, while == is closely related to the mathematical symbol '='.)

Answer: The syntax error seems to be caused by the not keyword, not (pun intended) the equality operator: True == not (5 > 7) # SyntaxError: invalid syntax  True == (not (5 > 7)) # True  The explanation can be found in the docs:  not has a lower priority than non-Boolean operators, so not a == b is interpreted as not (a == b), and a == not b is a syntax error.  Basically, the interpreter thinks you're comparing True to not.
Question: I am trying to combine two spatial xarray datasets using combine_by_coords. These two datasets are two tiles next to each other. So there are overlapping coordinates. In the overlapping regions, the variable values of one of the datasets is nan. I used the "combine_by_coords" with compat='no_conflicts' option. However, it returns the monotonic global indexes along dimension y error. It looks like it was an issue before but it was fixed (here). So I don't really know why I get this error. Here is an example (the netcdf tiles are here): import xarray as xr print(xr.__version__) >>>0.15.1 ds1=xr.open_dataset('Tile1.nc') ds2=xr.open_dataset('Tile2.nc') ds = xr.combine_by_coords([ds1,ds2], compat='no_conflicts') >>>... ValueError: Resulting object does not have monotonic global indexes along dimension y  Thanks

Answer: This isn't a bug, it's throwing the error it should be throwing given your input. However I can see how the documentation doesn't make it very clear as to why this is happening! combine_by_coords and combine_nested do two things: they concatenate (using xr.concat), and they merge (using xr.merge). merge groups variables of the same size, concat joins variables of different sizes onto the ends of one another. The concatenate step is never supposed to handle partially overlapping coordinates, and the combine functions therefore have the same restriction. That error is an explicit rejection of the input you gave it: "you gave me overlapping coordinates, I don't know how to concatenate those, so I'll reject them." Normally this makes sense - when the overlapping coordinates aren't NaNs then it's ambiguous as to which values to choose. In your case then you are asking it to perform a well-defined operation, and the discussion in the docs about merging overlapping coordinates here implies that compat='no_conflicts' would handle this situation. Unfortunately that's only for xr.merge, not xr.concat, and so it doesn't apply for combine_by_coords either. This is definitely confusing. It might be possible to generalise the combine functions to handle the scenario you're describing (where the overlapping parts of the coordinates are specified entirely by the non-NaN values). Please open an issue proposing this feature if you would like to see it. (Issue #3150 was about something else, an actual bug in the handling of "coordinate dimensions which do not vary between each dataset".) Instead, what you need to do is trim off the overlap first. That shouldn't be hard - presumably you know (or can determine) how big your overlap is, and all your NaNs are on one dataset. You just need to use the .isel() method with a slice. Once you've got rid of the overlapping NaNs then you should be able to combine it fine (and you shouldn't need to specify compat either). If you're using combine_by_coords as part of opening many files with open_mfdataset then it might be easier to write a trimming function which you apply first using the preprocess argument to open_mfdataset.
Question: A bit confused with automatisation of Sagemaker retraining the model. Currently I have a notebook instance with Sagemaker LinearLerner model making the classification task. So using Estimator I'm making training, then deploying the model creating Endpoint. Afterwards using Lambda function for invoke this endpoint, I add it to the API Gateway receiving the api endpoint which can be used for POST requests and sending back response with class. Now I'm facing with the problem of retraining. For that I use serverless approach and lambda function getting environment variables for training_jobs. But the problem that Sagemaker not allow to rewrite training job and you can only create new one. My goal is to automatise the part when the new training job and the new endpoint config will apply to the existing endpoint that I don't need to change anything in API gateway. Is that somehow possible to automatically attach new endpoint config with existing endpoint? Thanks

Answer: If I am understanding the question correctly, you should be able to use CreateEndpointConfig near the end of the training job, then use UpdateEndpoint: Deploys the new EndpointConfig specified in the request, switches to using newly created endpoint, and then deletes resources provisioned for the endpoint using the previous EndpointConfig (there is no availability loss). If the API Gateway / Lambda is routed via the endpoint ARN, that should not change after using UpdateEndpoint.
Question: Is it somehow possible to show syntax-highlighted python code in a webpage? I found this: <pre class="brush: python"> # python code here </pre>  However, it shows all the code in black. I want import to be orange, strings to be green. Is it possible to do this? Thank you!

Answer: If you wish to only display code, python in this case, consider using Github gist. You can then embed it using the 'embed' option on the top right corner. It will give you a script tag that you can copy and add to your webpage like so: <script src="https://gist.github.com/username/a39a422ebdff6e732753b90573100b16.js"></script>
Question: Pytest cov is not reading its setting from the pyproject.toml file. I am using nox, so I run the test with: python3 -m nox  It seems I have the same issue even without nox. In fact, after running a poetry install:  poetry run pytest --cov=src passes the test poetry run pytest --cov does not pass the test  In particular, when failing the test I have the following output (output is cut to the most important stuff): WARNING: Failed to generate report: No data to report. /Users/matteo/Library/Caches/pypoetry/virtualenvs/project-Nz69kfmJ-py3.7/lib/python3.7/site-packages/pytest_cov/plugin.py:271: PytestWarning: Failed to generate report: No data to report. self.cov_controller.finish() ---------- coverage: platform darwin, python 3.7.7-final-0 ----------- FAIL Required test coverage of 100.0% not reached. Total coverage: 0.00%  Code with a reproducible error here. To run it you'll need to install poetry and to install nox.

Answer: Turning the comment into an answer: Check the current treatment of the src directory. Right now, it seems to be a namespace package which is not what you intend. Either switch to the src layout: # pyproject.toml [tool.poetry] ... packages = [ { include = 'project', from = 'src' } ] [tool.coverage.run] ... source = ['project']  and fix the import in test_code.py: from src.project import code  to from project import code  or remove the src dir: rootdir ├── project │ └── __init__.py └── tests └── test_code.py  and fix the import in test_code.py.
Question: EDIT: As pointed out by some users, the request does not actually returns a JSON, but a string encoded JSON. The issue here is not actually parsing the JSON in python, but writing it in such a way that a request can be sent to the API. Therefore, it's not necessary to use the json python library. I'm using the various google APIs (slides, drive, sheets and such) with a python program. I am having issues accessing the properties in the JSON that the API requests return. Take a slides presentations().get request for example. It returns an instance of presentation. And I want to access its Slides property, which is an array of page objects. So I am trying the code slideInfo = SLIDES.presentations().get(presentationId=presId).execute() slideInfo = slideInfo.get(['slides'][0]['pageType'])  But I get an error message saying "TypeError: string indices must be integers" However, I thought that using brackets with a string was an acceptable replacement for accesing with a dot. In fact, I don't know how to translate this to dot accesors because it throws a syntax error since the keys have to be wrapped in quotes. 'slides'[0].'pageType'  throws syntax error because of the dot, and without the dot it doesnt work either

Answer: So with the JSON representation in the docs: { "presentationId": string, "pageSize": { object (Size) }, "slides": [ { object (Page) } ], "title": string, "masters": [ { object (Page) } ], "layouts": [ { object (Page) } ], "locale": string, "revisionId": string, "notesMaster": { object (Page) } }  You can access the slides using: slideInfo.get('slides') or slideInfo['slides'] So if you want to get the pageType of the first slide, it would be: slideInfo['slides'][0]['pageType']  or slideInfo.get('slides')[0].get('pageType')
Question: I read this code (given below) and my understanding was that, if a variable is declared global inside a function and if it is modified then it's value will change permanently. x = 15 def change(): global x x = x + 5 print("Value of x inside a function :", x) change() print("Value of x outside a function :", x)  Output: Value of x inside a function : 20 Value of x outside a function : 20  But the code below shows a different output. How is it that the value of x does not change inside the print("After making change: ", x)  and still remains 15 def add(): x = 15 def change(): global x x = 20 print("Before making changes: ", x) print("Making change") change() print("After making change: ", x) add() print("value of x",x)  Output: Before making changes: 15 Making change After making change: 15 value of x 20 

Answer: In add, x is not a global variable; it's local to add. You either need to make it global as well, so that add and change are referring to the same variable def add(): global x x = 15 def change(): global x x = 20 print("Before making changes: ", x) print("Making change") change() print("After making change: ", x) add() print("value of x",x) or you need to declare x in change as nonlocal, rather than global. def add(): x = 15 def change(): nonlocal x x = 20 print("Before making changes: ", x) print("Making change") change() print("After making change: ", x) add() print("value of x",x)
Question: I would like to create a new tensor in a validation_epoch_end method of a LightningModule. From the official docs (page 48) it is stated that we should avoid direct .cuda() or .to(device) calls:  There are no .cuda() or .to() calls. . . Lightning does these for you.  and we are encouraged to use type_as method to transfer to the correct device. new_x = new_x.type_as(x.type()) However, in a step validation_epoch_end I do not have any tensor to copy device from(by type_as method) in a clean way. My question is what should I do if I want to create a new tensor in this method and transfer it to the device where is the model? The only thing I can think of is to find a tensor in the outputs dictionary but it feels kinda messy: avg_loss = torch.stack([x['val_loss'] for x in outputs]).mean() output = self(self.__test_input.type_as(avg_loss))  Is there any clean way to achieve that?

Answer: did you check part 3.4 (page 34) in the doc you linked ?  LightningModules know what device they are on! construct tensors on the device directly to avoid CPU->Device transfer  t = tensor.rand(2, 2).cuda()# bad (self is lightningModule)t = tensor.rand(2,2, device=self.device)# good  I had a similar issue to create tensors this helped me. I hope it will help you too.
Question: Code block 1 using __init__ %%cython -3 cdef class c: cdef: int a str s def __init__(self): self.a=1 self.s="abc" def get_vals(self): return self.a,self.s m=c() print(m.get_vals())  Code block 2 using __cinit__ %%cython -3 cdef class c: cdef: int a str s def __cinit__(self): # cinit here self.a=1 self.s="abc" def get_vals(self): return self.a,self.s m=c() print(m.get_vals())   I tested both of these codes, and both run without error. In this case, what's the point of using __cinit__ instead of __init__?  I've read the official article, I got confused by one sentence:  If you need to pass a modified argument list to the base type, you will have to do the relevant part of the initialization in the __init__() method instead, where the normal rules for calling inherited methods apply.    What does "modified argument" mean? Here, why should I use init rather than cinit?

Answer: It's mainly about inheritance. Suppose I inherit from your class C: class D(C): def __init__(self): pass # oops forgot to call C.__init__ class E(C): def __init__(self): super().__init__(self) super().__init__(self) # called it twice  How __init__ ends up being called is entirely up to the classes that inherit from it. Bear in mind there may be multiple layers of inheritance. Additionally, a fairly common pattern for creating classes that wrap C/C++ objects is to create a staticmethod cdef function as an alternative constructor: cdef class C: def __cinit__(self): print("In __cinit__") @staticmethod cdef make_from_ptr(void* x): val = C.__new__(C) # do something with pointer return val  In this case again, __init__ typically isn't called. In contrast __cinit__ is guaranteed to be called exactly once and this happens automatically by Cython at an early stage of the process. This is most important when you have cdef attributes (such as C pointers) that your class relies on being initialized. It would be impossible for a Python derived class to even set these up, but __cinit__ can ensure that they are. In your case it probably doesn't matter - use whichever you're happy with.  In terms of "modified arguments" it's saying you can't replicate this with __cinit__: class NameValue: def __init__(self, name, value): self.name = name self.value = value class NamedHelloPlus1(NamedValue): def __init__(self, value): super().__init__("Hello", value+1)  i.e. NamedHelloPlus1 controls what arguments NamedValue gets. With __cinit__ Cython all the calls to __cinit__ receive exactly the same arguments (because Cython arranges the call - you cannot call it manually).
Question: Context I had a dockerfile based on postgres:11-alpine that was working in the past (probably a few months since it was last built) with the following definition: FROM postgres:11-alpine RUN apk update # install aws cli # taken from: https://github.com/anigeo/docker-awscli/blob/master/Dockerfile RUN \ apk -Uuv add groff less python py-pip && \ pip install awscli && \ apk --purge -v del py-pip && \ rm /var/cache/apk/*  I recently tried to rebuild it before upgrading to postgres 12, but the image build failed with:  ERROR: unsatisfiable constraints: python (missing): required by: world[python]  I guess the python package is gone now because YOLO? Whatever, I tried to upgrade to python3 by changing the docker file to: RUN \ apk -Uuv add groff less python3 py-pip && \ pip install awscli && \ apk --purge -v del py-pip && \ rm /var/cache/apk/*  This looked like it worked, but then when running the aws command it failed with error: ModuleNotFoundError: No module named 'six'   Question How to fix this so the awscli will not give the error No module named 'six'?

Answer: The problem seems to actually be caused by deleting py-pip. As far as I know, the aim of the apk del was to reduce the size of the final docker image. I'm not sure why deleting py-pip used to work when the file was using the python package. So the following now seems to be working: RUN \ apk -Uuv add groff less python3 py-pip && \ pip install awscli && \ rm /var/cache/apk/*
Question: I have a problem, I'm trying to build my own class to put into a pipeline in python, but it doesn't work. The problem I am trying to solve is a multiclass classification problem. What I want to do this to add a step in the pipeline to detect and remove outliers. I found this detect and remove outliers in pipeline python which is very similar to what I did. This is my class: from sklearn.neighbors import LocalOutlierFactor from sklearn.base import BaseEstimator, TransformerMixin import numpy as np class OutlierExtraction(BaseEstimator, TransformerMixin): def __init__(self, **kwargs ): self.kwargs = kwargs def transform(self, X, y): """ X should be of shape (n_samples, n_features) y should be of shape (n_samples,) """ lof = LocalOutlierFactor(**self.kwargs) lof.fit(X) nof = lof.negative_outlier_factor_ return X[nof > np.quantile(nof, 0.95), :], y[nof > np.quantile(nof, 0.95)] def fit(self, X, y = None): return self  But i get this error in fit_transform return self.fit(X, y, **fit_params).transform(X) TypeError: transform() missing 1 required positional argument: 'y' The following code is the code i use to call this class: scaler = preprocessing.RobustScaler() outlierExtractor = OutlierExtraction() pca = PCA() classfier = svm.SVC() pipeline = [('scaler', scaler), ('outliers', outlierExtractor), ('reduce_dim', pca), ('classfier', classfier)] pipe = Pipeline(pipeline) params = { 'reduce_dim__n_components': [5, 15], 'classfier__kernel': ['rbf'], 'classfier__gamma': [0.1], 'classfier__C': [1], 'classfier__decision_function_shape':['ovo']} my_scoring = 'f1_macro' n_folds = 5 gscv = GridSearchCV(pipe, param_grid=params, scoring=my_scoring, n_jobs=-1, cv=n_folds, refit=True) gscv.fit(train_x, train_y) 

Answer: The error is because the transform method def transform(self, X, y) requires both X and y to be passed in, but whatever is calling it is only passing X. (I can't see where it's called from in your code so assume it's being called by the underlying library). I don't know if making y optional (def transform(self, X, y=None) and modifying your method would work in this case. Otherwise, you'll have to figure out how to get the calling code to pass y, or provide it another way. I'm not familiar with the library, but looking at the source code shows that transform() should only take a single parameter X:  if y is None: # fit method of arity 1 (unsupervised transformation) return self.fit(X, **fit_params).transform(X) else: # fit method of arity 2 (supervised transformation) return self.fit(X, y, **fit_params).transform(X)
Question: I have the following pandas dataframe : df = pd.DataFrame([ ['A', 2017, 1], ['A', 2019, 1], ['B', 2017, 1], ['B', 2018, 1], ['C', 2016, 1], ['C', 2019, 1], ], columns=['ID', 'year', 'number'])  and am looking for the most efficient way to fill the missing years with a default value of 0 for the column number The expected output is:  ID year number 0 A 2017 1 1 A 2018 0 2 A 2019 1 3 B 2017 1 4 B 2018 1 5 C 2016 1 6 C 2017 0 7 C 2018 0 8 C 2019 1  The dataframe that I have is relatively big, so I am looking for an efficient solution. Edit: This is the code that I have so far: min_max_dict = df[['ID', 'year']].groupby('ID').agg([min, max]).to_dict('index') new_ix = [[], []] for id_ in df['ID'].unique(): for year in range(min_max_dict[id_][('year', 'min')], min_max_dict[id_][('year', 'max')]+1): new_ix[0].append(id_) new_ix[1].append(year) df.set_index(['ID', 'year'], inplace=True) df = df.reindex(new_ix, fill_value=0).reset_index()  Result  ID year number 0 A 2017 1 1 A 2018 0 2 A 2019 1 3 B 2017 1 4 B 2018 1 5 C 2016 1 6 C 2017 0 7 C 2018 0 8 C 2019 1 

Answer: A slightly faster approach rather than using explode is to use pd.Series constructor. And you can use .iloc if years are already sorted from earliest to latest. idx = df.groupby('ID')['year'].apply(lambda x: pd.Series(np.arange(x.iloc[0], x.iloc[-1]+1))).reset_index() df.set_index(['ID','year']).reindex(pd.MultiIndex.from_arrays([idx['ID'], idx['year']]), fill_value=0).reset_index()  Output:  ID year number 0 A 2017 1 1 A 2018 0 2 A 2019 1 3 B 2017 1 4 B 2018 1 5 C 2016 1 6 C 2017 0 7 C 2018 0 8 C 2019 1
Question: I want to read a JSON file with Python : Here is part of my JSON file : { "Jointure":[ { "IDJointure":1, "societe":"S.R.T.K", "date":"2019/01/01", "heure":"05:47:00"}, { "IDJointure":2, "societe":"S.R.T.K", "date":"2019/01/01", "heure":"05:50:00"}]}  This is the code : import json data = json.loads('Data2019.json') for i in data['Jointure']: print(i)  But, here is the error that was displayed Traceback (most recent call last): File "C:\Users\HP\Desktop\readJSON.py", line 4, in <module> data = json.loads('Data2019.json') File "C:\Users\HP\AppData\Local\Programs\Python\Python38\lib\json\__init__.py", line 357, in loads return _default_decoder.decode(s) File "C:\Users\HP\AppData\Local\Programs\Python\Python38\lib\json\decoder.py", line 337, in decode obj, end = self.raw_decode(s, idx=_w(s, 0).end()) File "C:\Users\HP\AppData\Local\Programs\Python\Python38\lib\json\decoder.py", line 355, in raw_decode raise JSONDecodeError("Expecting value", s, err.value) from None json.decoder.JSONDecodeError: Expecting value: line 1 column 1 (char 0) >>> 

Answer: Try pandas   import pandas as pd patients_df = pd.read_json('E:/datasets/patients.json') patients_df.head()
Question: If I have a list say l = [1, 8, 8, 8, 1, 3, 3, 8] and it's guaranteed that every element occurs an even number of times, how do I make a list with all elements of l now occurring n/2 times. So since 1 occurred 2 times, it should now occur once. Since 8 occurs 4 times, it should now occur twice. Since 3 occurred twice, it should occur once. So the new list will be something like k=[1,8,8,3] What is the fastest way to do this? I did list.count() for every element but it was very slow.

Answer: If order isn't important, a way would be to get the odd or even indexes only after a sort. Those lists will be the same so you only need one of them. l = [1,8,8,8,1,3,3,8] l.sort() # Get all odd indexes odd = l[1::2] # Get all even indexes even = l[::2] print(odd) print(odd == even)  Result: [1, 3, 8, 8] True