eshangj/stackoverflow_q_and_a_sample huggingface dataset:
Question: I have imported my entire project into docker, and I am getting a   ModuleNotFoundError  from one of the modules I have created. FROM python:3.8 WORKDIR /workspace/ COPY . /workspace/ RUN pip install pipenv RUN pipenv install --deploy --ignore-pipfile #EXPOSE 8000 #CMD ["pipenv", "run", "python", "/workspace/bin/web.py"]  I tried looking around for answers, but I cannot seem to get it working. commands: docker build -t atletico . docker run -p 8000:8000 atletico  Docker Build: https://pastebin.com/FXMrY2En Traceback (most recent call last): File "/workspace/bin/web.py", line 3, in <module> from bin.setup import setup_app ModuleNotFoundError: No module named 'bin'  A copy of my directory: ├── Dockerfile ├── Pipfile ├── Pipfile.lock ├── README.md ├── bin │ ├── __init__.py │ ├── __pycache__ │ │ └── web.cpython-38.pyc │ ├── setup.py │ └── web.py ├── docker-compose.yml ├── frio │ ├── __init__.py │ ├── __pycache__ │ │ └── __init__.cpython-38.pyc │ ├── app_events.py │ └── config.py ├── routes │ ├── __init__.py  docker-compose.yml: version: '3' services: db: image: postgres:12 ports: - "5432:5432" environment: - POSTGRES_USER=postgres - POSTGRES_PASSWORD=postgres - POSTGRES_DB=test_db redis: image: "redis:alpine" web: env_file: - .env.local build: . ports: - "8000:8000" volumes: - .:/workspace depends_on: - db - redis command: "pipenv run python /workspace/bin/web.py" 

Answer: So I finally fixed the issue. For those who may be wondering how was it that I fixed it. You need to define a PYTHONPATH environment variable either in the Dockerfile or docker-compose.yml.
Question: I have a pandas series,  and this is the graph:  I want to fill the area under the curve. The problem is that calling plt.fill(y)  outputs:  As seen in other answers, this is because we need to send a polygon to the function, so we have to add a (0,0) point. (And a (lastPoint, 0), but in this case it's not necessary). However, the proposed solution is writing the following code: plt.fill([0]+[*range(0,len(y))], [0]+pd.Series.tolist(y))  I refuse to believe this is the best solution. The code is horrible, not at all easy to read, and I am losing information (no dates on x axis):  Furthermore, if I call both plot and fill (to have the red line on the top), an error occurs: /usr/local/anaconda3/lib/python3.7/site-packages/matplotlib/dates.py in refresh(self) 1446 def refresh(self): 1447 'Refresh internal information based on current limits.' -> 1448 dmin, dmax = self.viewlim_to_dt() 1449 self._locator = self.get_locator(dmin, dmax) 1450 /usr/local/anaconda3/lib/python3.7/site-packages/matplotlib/dates.py in viewlim_to_dt(self) 1197 'often happens if you pass a non-datetime ' 1198 'value to an axis that has datetime units' -> 1199 .format(vmin)) 1200 return num2date(vmin, self.tz), num2date(vmax, self.tz) 1201 ValueError: view limit minimum -36868.15 is less than 1 and is an invalid Matplotlib date value. This often happens if you pass a non-datetime value to an axis that has datetime units  So I was hoping somebody could help me write better code and resolve this issue. I think matplotlib should add a function fill_area or similar. What do you guys think about this?

Answer: There is such a function: matplotlib.pyplot.fill_between() import matplotlib.pyplot as plt plt.plot(y, c='red') plt.fill_between(y.index, y, color='blue', alpha=0.3)
Question: Is it possible to add a default value to ArrayField? I tried to do this for email field, but this did not work: constants.py: ORDER_STATUS_CHANGED = 'order_status_changed' NEW_SIGNAL = 'new_signal' NOTIFICATION_SOURCE = ( (ORDER_STATUS_CHANGED, 'Order Status Changed'), (NEW_SIGNAL, 'New Signal'), )  models.py: from notifications import constants from django.contrib.postgres.fields import ArrayField class NotificationSetting(models.Model): user = models.OneToOneField(User, on_delete=models.CASCADE, primary_key=True, related_name='notification_setting') telegram = ArrayField(models.CharField( choices= constants.NOTIFICATION_SOURCE, max_length=30 ), default=list) email = ArrayField(models.CharField( choices= constants.NOTIFICATION_SOURCE, max_length=16 ), default=list(dict(constants.NOTIFICATION_SOURCE).keys())) class Meta: db_table = 'notification_settings' def __str__(self): return f'Notification setting for user {self.user}'  And override the save method of the model would be bad practice, I think. The problem is that in the django admin site I see that the default values did not count when the object was created. (UPD. Maibe i have problem with my custom ChoiseArrayField widged) And i get this mesagge: WARNINGS: notifications.NotificationSetting.email: (postgres.E003) ArrayField default should be a callable instead of an instance so that it's not shared between all field instances. HINT: Use a callable instead, e.g., uselistinstead of[]``

Answer: The default property on an ArrayField should be a callable. You can read more about that here: https://docs.djangoproject.com/en/3.0/ref/contrib/postgres/fields/. What you are getting by placing directly there list(dict(constants.NOTIFICATION_SOURCE).keys()) is just a warning so it should still add the defaults to the field. By placing this default directly there it will put in the migrations the following thing and the values will be shared across all field instances:  default=['order_status_changed', 'new_signal']  To get rid of the warning you should create a function that returns the default value: def get_email_default(): return list(dict(constants.NOTIFICATION_SOURCE).keys())  and put the function as the default to the field: email = ArrayField(models.CharField( choices= constants.NOTIFICATION_SOURCE, max_length=16 ), default=get_email_default)  By doing this the warning will be gone and from the function you can have logic for choosing the default value. After doing this, in the migrations the default value will look like this: default=my_model.models.get_email_default
Question: Can anyone help me solve this issue? ImportError: dlopen(/Users/......./venv/lib/python3.6/site-packages/recordclass/mutabletuple.cpython-36m-darwin.so, 2): Symbol not found: __PyEval_GetBuiltinId Referenced from: /Users/......./venv/lib/python3.6/site-packages/recordclass/mutabletuple.cpython-36m-darwin.so Expected in: flat namespace in /Users/......../venv/lib/python3.6/site-packages/recordclass/mutabletuple.cpython-36m-darwin.so  I'm using a Mac if that's of any relevance

Answer: I couldn't quite figure out what the issue was but I'm assuming __PyEval_GetBuiltinId was broken/uninstalled. So all I did to fix this was pip uninstall recordclass and then pip install --no-cache-dir recordclass and it seemed to have worked
Question: An mwe is as follows: import torch import torch.nn as nn class model(nn.Module): def __init__(self): super(model,self).__init__() self.mat = torch.randn(2,2) def forward(self,x): print('self.mat.device is',self.mat.device) x = torch.mv(self.mat,x) return x device = torch.device('cuda' if torch.cuda.is_available() else 'cpu') m = model() m.to(device) x = torch.tensor([2.,1.]) x = x.to(device) m(x)  The output is self.mat.device is cpu  and right after that comes Traceback (most recent call last): File "Z:\cudatest.py", line 21, in <module> print(m(x)) File "E:\Python37\lib\site-packages\torch\nn\modules\module.py", line 532, in __call__ result = self.forward(*input, **kwargs) File "Z:\cudatest.py", line 11, in forward x = torch.mv(self.mat,x) RuntimeError: Expected object of device type cuda but got device type cpu for argument #1 'self' in call to _th_mv  The code works fine if I set device = torch.device('cpu'). It seems that the problem is model.mat is not moved to GPU even after m.to(device) is called. Why doesn't whis work? How can I fix this? Please note the following:  Even though this particular example can be fixed by using self.mat = nn.Linear(2,2) and x = self.mat(x) instead, in my original program, I need a temporary tensor to store some data in forward() that is also used in some arithmetics. How can I construct such a tensor and send it to GPU when calling m.to(device) It is not known in advance whether the computer has GPU or not. Therefore, writing self.mat = self.mat.cuda() is not a good solution for my case. 

Answer: pytorch apply Module's methods such as .cpu(), .cuda() and .to() only to sub-modules, parameters and buffers, but NOT to regular class members. pytorch has no way of knowing that self.mat, in your case, is an actual tensor that should be moved around. Once you decide if your mat should be a parameter or a buffer, simply register it accordingly, e.g. class model(nn.Module): def __init__(self): super(model,self).__init__() self.register_buffer(name='mat', tensor=torch.randn(2,2))
Question: I've written tests to be run by pytest in a vscode project. The configuration file .vscode/settings.json allow passing additional command line parameters to pytest using:  "python.testing.pytestArgs": [ "test/", "--exitfirst", "--verbose" ],  How can I also pass custom script arguments to the test script itself? like invoking pytest from the command line as: pytest --exitfirst --verbose test/ --test_arg1 --test_arg2 

Answer: After much experimentation I finally found how to do it. What I needed was to pass user name and password to my script in order to allow the code to log into a test server. My test looked like this: my_module_test.py import pytest import my_module def login_test(username, password): instance = my_module.Login(username, password) # ...more...  conftest.py import pytest def pytest_addoption(parser): parser.addoption('--username', action='store', help='Repository user') parser.addoption('--password', action='store', help='Repository password') def pytest_generate_tests(metafunc): username = metafunc.config.option.username if 'username' in metafunc.fixturenames and username is not None: metafunc.parametrize('username', [username]) password = metafunc.config.option.password if 'password' in metafunc.fixturenames and password is not None: metafunc.parametrize('password', [password])  Then in my settings file I can use: .vscode/settings.json { // ...more... "python.testing.autoTestDiscoverOnSaveEnabled": true, "python.testing.unittestEnabled": false, "python.testing.nosetestsEnabled": false, "python.testing.pytestEnabled": true, "python.testing.pytestArgs": [ "--exitfirst", "--verbose", "test/", "--username=myname", "--password=secret", // ...more... ], }  An alternative way is to use pytest.ini file instead: pytest.ini [pytest] junit_family=legacy addopts = --username=myname --password=secret
Question: Grid search is a way to find the best parameters for any model out of the combinations we specify. I have formed a grid search on my model in the below manner and wish to find best parameters identified using this gridsearch. from sklearn.model_selection import GridSearchCV # Create the parameter grid based on the results of random search param_grid = { 'bootstrap': [True],'max_depth': [20,30,40, 100, 110], 'max_features': ['sqrt'],'min_samples_leaf': [5,10,15], 'min_samples_split': [40,50,60], 'n_estimators': [150, 200, 250] } # Create a based model rf = RandomForestClassifier() # Instantiate the grid search model grid_search = GridSearchCV(estimator = rf, param_grid = param_grid, cv = 3, n_jobs = -1, verbose = 2)  Now I want to find the best parameters of the gridsearch as the output grid_search.best_params_  Error: ----> grid_search.best_params_ AttributeError: 'GridSearchCV' object has no attribute 'best_params_'  What am I missing?

Answer: You cannot get best parameters without fitting the data.  Fit the data grid_search.fit(X_train, y_train)  Now find the best parameters. grid_search.best_params_  grid_search.best_params_ will work after fitting on X_train and y_train.
Question: so I've been trying to code a Python script, which takes an image as input and then cuts out a rectangle with a specific background color. However, what causes a problem for my coding skills, is that the rectangle is not on a fixed position in every image (the position will be random).  I do not really understand how to manage the numpy functions. I also read something about OpenCV, but I'm totally new to it. So far I just cropped the images through the ".crop" function, but then I would have to use fixed values. This is how the input image could look and now I would like to detect the position of the yellow rectangle and then crop the image to its size. Help is appreciated, thanks in advance.   Edit: @MarkSetchell's way works pretty good, but found a issue for a different test picture. The problem with the other picture is that there are 2 small pixels with the same color at the top and the bottom of the picture, which cause errors or a bad crop.

Answer: Updated Answer I have updated my answer to cope with specks of noisy outlier pixels of the same colour as the yellow box. This works by running a 3x3 median filter over the image first to remove the spots: #!/usr/bin/env python3 import numpy as np from PIL import Image, ImageFilter # Open image and make into Numpy array im = Image.open('image.png').convert('RGB') na = np.array(im) orig = na.copy() # Save original # Median filter to remove outliers im = im.filter(ImageFilter.MedianFilter(3)) # Find X,Y coordinates of all yellow pixels yellowY, yellowX = np.where(np.all(na==[247,213,83],axis=2)) top, bottom = yellowY[0], yellowY[-1] left, right = yellowX[0], yellowX[-1] print(top,bottom,left,right) # Extract Region of Interest from unblurred original ROI = orig[top:bottom, left:right] Image.fromarray(ROI).save('result.png')  Original Answer Ok, your yellow colour is rgb(247,213,83), so we want to find the X,Y coordinates of all yellow pixels: #!/usr/bin/env python3 from PIL import Image import numpy as np # Open image and make into Numpy array im = Image.open('image.png').convert('RGB') na = np.array(im) # Find X,Y coordinates of all yellow pixels yellowY, yellowX = np.where(np.all(na==[247,213,83],axis=2)) # Find first and last row containing yellow pixels top, bottom = yellowY[0], yellowY[-1] # Find first and last column containing yellow pixels left, right = yellowX[0], yellowX[-1] # Extract Region of Interest ROI=na[top:bottom, left:right] Image.fromarray(ROI).save('result.png')    You can do the exact same thing in Terminal with ImageMagick: # Get trim box of yellow pixels trim=$(magick image.png -fill black +opaque "rgb(247,213,83)" -format %@ info:) # Check how it looks echo $trim 251x109+101+220 # Crop image to trim box and save as "ROI.png" magick image.png -crop "$trim" ROI.png  If still using ImageMagick v6 rather than v7, replace magick with convert.
Question: I have list column names. columns = ['home','house','office','work']  and I would like to pass that list values as columns name in "select" dataframe. I have tried it... df_tables_full = df_tables_full.select('time_event','kind','schema','table',columns)  but I have received error below.. TypeError: Invalid argument, not a string or column: ['home', 'house', 'office', 'work'] of type <class 'list'>. For column literals, use 'lit', 'array', 'struct' or 'create_map' function.  Can you have any ideia? Thank you guys!

Answer: Use * before columns to unnest columns list and use in .select. columns = ['home','house','office','work'] #select the list of columns df_tables_full.select('time_event','kind','schema','table',*columns).show() df_tables_full = df_tables_full.select('time_event','kind','schema','table',*columns)
Question: I am trying to use pySerial==3.4, and find the documentation on serial.Serial.flush() rather lacking:  Flush of file like objects. In this case, wait until all data is written.  Source  Questions  What is a "file like object"? What is being flushed? When would one use flush as opposed to just individually resetting the input/output buffers?  serial = Serial("COM3") # Option 1 serial.flush() # Option 2 serial.reset_input_buffer() serial.reset_output_buffer()   Relevant Questions  using pyserial flush method 

Answer: It looks like this: What is a "file like object"? What is exactly a file-like object in Python?  file-like objects are mainly StringIO objects, connected sockets and well.. actual file objects. If everything goes fine, urllib.urlopen() also returns a file-like objekt supporting the necessary methods. file-like object A synonym for file object. file object An object exposing a file-oriented API (with methods such as read() or write()) to an underlying resource. Depending on the way it was created, a file object can mediate access to a real on-disk file or to another type of storage or communication device (for example standard input/output, in-memory buffers, sockets, pipes, etc.). File objects are also called file-like objects or streams. There are actually three categories of file objects: raw binary files, buffered binary files and text files. Their interfaces are defined in the io module. The canonical way to create a file object is by using the open() function.  io — Core tools for working with streams  The io module provides Python’s main facilities for dealing with various types of I/O. There are three main types of I/O: text I/O, binary I/O and raw I/O. These are generic categories, and various backing stores can be used for each of them. A concrete object belonging to any of these categories is called a file object. Other common terms are stream and file-like object.  What is being flushed? The data held in the output buffer. When would one use flush as opposed to just individually resetting the input/output buffers? There is data that has been output(write()), and it will be called before closing. flush() has nothing to do with input buffer or reset_input_buffer(). flush() has a different function than reset_output_buffer(). flush() sends all the data in the output buffer to the peer, while reset_output_buffer() discards the data in the output buffer.  reset_output_buffer() Clear output buffer, aborting the current output and discarding all that is in the buffer. Note, for some USB serial adapters, this may only flush the buffer of the OS and not all the data that may be present in the USB part.
Question: Reading http://doc.pytest.org/en/latest/example/markers.html I see the example of including or excluding certain python tests based on a mark. Including: pytest -v -m webtest  Excluding: pytest -v -m "not webtest"  What if I would like to specify several marks for both include and exclude?

Answer: Use and/or to combine multiple markers, same as for -k selector. Example test suite: import pytest @pytest.mark.foo def test_spam(): assert True @pytest.mark.foo def test_spam2(): assert True @pytest.mark.bar def test_eggs(): assert True @pytest.mark.foo @pytest.mark.bar def test_eggs2(): assert True def test_bacon(): assert True  Selecting all tests marked with foo and not marked with bar $ pytest -q --collect-only -m "foo and not bar" test_mod.py::test_spam test_mod.py::test_spam2  Selecting all tests marked neither with foo nor with bar $ pytest -q --collect-only -m "not foo and not bar" test_mod.py::test_bacon  Selecting tests that are marked with any of foo, bar $ pytest -q --collect-only -m "foo or bar" test_mod.py::test_spam test_mod.py::test_spam2 test_mod.py::test_eggs test_mod.py::test_eggs2
Question: I have a dataframe that looks like this  Australia Austria United Kingdom Vietnam date 2020-01-30 9 0 1 2 2020-01-31 9 9 4 2  I would like to crate a new dataframe that inclues countries that have sum of their column > 4 and I do it df1 = df[[i for i in df.columns if int(df[i].sum()) > 4]]  this gives me   Australia Austria United Kingdom date 2020-01-30 9 0 1 2020-01-31 9 9 4  I now would like to sort the countries based on the sum of their column and than take the first 2  Australia Austria date 2020-01-30 9 0 2020-01-31 9 9  I know I have to use sort_values and tail. I just can't workout how

Answer: IIUC, you can do: s = df.sum() df[s.sort_values(ascending=False).index[:2]]  Output:  Australia Austria date 2020-01-30 9 0 2020-01-31 9 9
Question: Code: from google.cloud import storage client = storage.Client() bucket = ['symbol_wise_nse', 'symbol_wise_final'] for i in bucket: if client.get_bucket(i).exists(): BUCKET = client.get_bucket(i)  if the bucket exists i want to do client.get_bucket. How to check whether the bucket exists or not?

Answer: There is no method to check if the bucket exists or not, however you will get an error if you try to access a non existent bucket. I would recommend you to either list the buckets in the project with storage_client.list_buckets() and then use the response to confirm if the bucket exists in your code, or if you wish to perform the client.get_bucket in every bucket in your project, you can just iterate through the response directly. Hope you find this information useful
Question: I have a list with name of organizations like this:  name = ['ALPHABET INC', 'AMAZON COM INC', 'APPLE INC',....]  and another list of cu values like this:  cu = ['02079K305', '023135106', '037833100',....]  When i'm trying to convert it to dataframe it's giving me error message saying, "ValueError: 2 columns passed, passed data had 1 columns" My code to convert list ot dataframe: df = pd.DataFrame([name, cu], columns=['name of issuer', 'cusip'])  Where am i going wrong? Thanks in advance!

Answer: I think simpliest is create dictionaries: df = pd.DataFrame({'name of issuer': name, 'cusip':cu})  Your solution is possible with zip, in last version of pandas should be omit list: df = pd.DataFrame(list(zip(name, cu)), columns=['name of issuer', 'cusip'])   print (df) name of issuer cusip 0 ALPHABET INC 02079K305 1 AMAZON COM INC 023135106 2 APPLE INC 037833100
Question: I created a Dockerfile which sets up python environment so that when people run the image in their host, they can run my python file without installing multiple python packages themselves.  The problem is after I build and run the image, the container stopped immediately (because it completed).  How can I keep the container running forever? My understanding is after people pull and run my image, they can start to run python file by running "python file.python". my Dockerfile looks like this (may not be correct. I am still learning): FROM python:3-alpine ADD . /app WORKDIR /app RUN pip install configparser 

Answer: From HOW TO KEEP DOCKER CONTAINERS RUNNING, we can know that docker containers, when run in detached mode (the most common -d option), are designed to shut down immediately after the initial entrypoint command (program that should be run when container is built from image) is no longer running in the foreground. So to keep docker container run even the inside program has done  just add CMD tail -f /dev/null as last line to dockerfile  What's more important is that we should understand what's docker is intended for and how to use it properly. Try to use docker as environment foundation for applications in host machine is not a good choice, docker is designed to run applications environment-independent. Applications should be placed into docker image via docker build and run in docker container in runtime.