eshangj/stackoverflow_q_and_a_sample huggingface dataset:
Question: In a module, I have two tests: @pytest.fixture def myfixture(request): prepare_stuff() yield 1 clean_stuff() # time.sleep(10) # in doubt, I tried that, did not help def test_1(myfixture): a = somecode() assert a==1 def test_2(myfixture): b = somecode() assert b==1  case 1 When these two tests are executed individually, all is ok, i.e. both pytest ./test_module.py:test_1  and immediately after: pytest ./test_module.py:test_2  run until completion and pass with success. case 2 But: pytest ./test_module.py -k "test_1 or test_2"  reports: collected 2 items test_module.py .  and hangs forever (after investigation: test_1 completed successfully, but the second call to prepare_stuff hangs). question In my specific setup prepare_stuff, clean_stuff and somecode are quite evolved, i.e. they create and delete some shared memory segments, which when done wrong can results in some hanging. So some issue here is possible. But my question is: are there things occurring between two calls of pytest (case 1) that do not occur between the call of test_1 and test_2 from the same "pytest process" (case 2), which could explain why "case 1" works ok while "case 2" hangs between test_1 and test_2 ? If so, is there a way to "force" the same "cleanup" to occur between test_1 and test_2 for "case 2" ? Note: I already tried to specify the scope of "myfixture" to "function", and also double checked that "clean_stuff" is called after "test_1", even in "case 2".

Answer: Likely something is happening in your prepare_stuff and/or clean_stuff functions. When you run tests as: pytest ./test_module.py -k "test_1 or test_2" they are running in the same execution context, same process, etc. So, if, for example, clean_stuff doesn't do a proper clean up then execution of a next test can fail. When you run tests as: pytest ./test_module.py:test_1 pytest ./test_module.py:test_2  they are running in different execution contexts, i.e. they start in an absolutely clean environment, and, unless you're modifying some external resources, you can easily remove clean_stuff in this case and they will pass anyway. To rule out pytest issue just try to run:  prepare_stuff() a = somecode() assert a==1 clean_stuff() prepare_stuff() b = somecode() assert b==1 clean_stuff()  I'm pretty sure you'll have the same problem, which would confirm that the issue is in your code, but not in the pytest.
Question: I have the following function: import pandas as pd def eq(left: pd.Timestamp, right: pd.Timestamp) -> bool: return left == right  I get the following error when I run it through Mypy:  error: Returning Any from function declared to return "bool"  I believe this is because Mypy doesn't know about pd.Timestamp so treats it as Any. (Using the Mypy reveal_type function shows that Mypy treats left and right as Any.) What is the correct way to deal with this to stop Mypy complaining?

Answer: you can cast it as a bool. import pandas as pd def eq(left: pd.Timestamp, right: pd.Timestamp) -> bool: return bool(left == right)  if mypy doesn't like that you can import cast from typing and use that to cast it to a bool. import pandas as pd from typing import cast def eq(left: pd.Timestamp, right: pd.Timestamp) -> bool: result = bool(left == right) return cast(bool, result)
Question: As finally OpenAI opened the GPT-3 related API publicly, I am playing with it to explore and discover his potential. I am trying the Answer API, the simple example that is in the documentation: https://beta.openai.com/docs/guides/answers I upload the .jsonl file as indicated, and I can see it succesfully uploaded with the openai.File.list() api. When I try to use it, unfortunately, I always get the same error: >>> openai.File.create(purpose='answers', file=open('example.jsonl') ) <File file id=file-xxx at 0x7fbc9eca5e00> JSON: { "bytes": 140, "created_at": 1637597242, "filename": "example.jsonl", "id": "file-xxx", "object": "file", "purpose": "answers", "status": "uploaded", "status_details": null } #Use the file in the API: openai.Answer.create( search_model="ada", model="curie", question="which puppy is happy?", file="file-xxx", examples_context="In 2017, U.S. life expectancy was 78.6 years.", examples=[["What is human life expectancy in the United States?", "78 years."]], max_rerank=10, max_tokens=5, stop=["\n", "
"] ) <some exception, then> openai.error.InvalidRequestError: File is still processing. Check back later.  I have waited several hours, and I do not think this content deserve such a long wait... Do you know if it is a normal behaviour, or if I miss something? Thanks

Answer: After a few hours (the day after) the file metadata status changed from uploaded to processed and the file could be used in the Answer API as stated in the documentation. I think this need to be better documented in the original OpenAI API reference.
Question: I have been provided with a pem certificate to authenticate with a third party. Authenticating using certificates is a new concept for me. Inside are two certificates and a private key. The issuer has advised they do not support SSL verification but use TLS(1.1/1.2). I have run a script as below: import requests as req import json url = 'https://url.com/call' certificate_file = "C:/certs/cert.pem" headers = {"Content-Type": "application/json"} req_body ={ "network":{ "network_id": 12345 }, "branch":{ "branch_id": 12345, }, "export_period":{ "start_date_time": "16-11-2021 00:00:00", "end_date_time": "17-11-2021 00:00:00" } } jsonObject = json.dumps(req_body) response = req.post(url,headers=headers,params=jsonObject,verify=certificate_file)  I'm getting the following error:  SSLError: HTTPSConnectionPool(host='url.com, port=443): Max retries exceeded with url: /call?%7B%22network%22:%20%7B%22network_id%22:%2012345%7D,%20%22branch%22:%20%7B%22branch_id%22:%2012345%7D,%20%22export_period%22:%20%7B%22start_date_time%22:%20%2216-11-2021%2000:00:00%22,%20%22end_date_time%22:%20%2217-11-2021%2000:00:00%22%7D%7D (Caused by SSLError(SSLCertVerificationError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get local issuer certificate (_ssl.c:1123)')))  Would appreciate guidance, my gut says I should be doing something specific for TLS hence the SSL error.

Answer: The issuer is using an up to date version of HTTPS - SSL is the commonly used term but TLS is the more correct one. Sounds like their setup is correct - meaning you need to call it with a trusted client certificate, as part of an HTTPS request. I would recommend doing it with the curl tool first so that you can verify that the API works as expected. curl -s -X GET "https://api.example.com/test" \ --cert ./certs/example.client.pem \ --key ./certs/example.client.key \ --cacert ./certs/ca.pem \ -H "Content-Type: application/json"  Split the certs into separate files as above. Sounds like one of them is a root certificate authority that you need to tell the client tech stack to trust - for curl this is done using the cacert parameter as above. Once this is working you can follow the same approach in the Python requests library. I believe this uses cert and verify parameters like this. So it looks like your code is not far off. result = requests.get( 'https://api.example.com', cert=('example.pem', 'example.key'), verify='ca.pem')  MORE ABOUT MUTUAL TLS Out of interest, if you ever want to demystify Mutual TLS and understand more, have a look at these advanced Curity resources:  Mutual TLS Secured API Code Sample  These include an OpenSSL Script you can run, to see what the separated certificate files should look like.
Question: Given an sklearn tranformer t, is there a way to determine whether t changes columns/column order of any given input dataset X, without applying it to the data? For example with t = sklearn.preprocessing.StandardScaler there is a 1-to-1 mapping between the columns of X and t.transform(X), namely X[:, i] -> t.transform(X)[:, i], whereas this is obviously not the case for sklearn.decomposition.PCA. A corollary of that would be: Can we know, how the columns of the input will change by applying t, e.g. which columns an already fitted sklearn.feature_selection.SelectKBest chooses. I am not looking for solutions to specific transformers, but a solution applicable to all or at least a wide selection of transformers. Feel free to implement your own Pipeline class or wrapper if necessary.

Answer: Not all your "transformers" would have the .get_feature_names_out method. Its implementation is discussed in the sklearn github. In the same link, you can see there is, to quote @thomasjpfan, a _OneToOneFeatureMixin class used by transformers with a simple one-to-one correspondence between input and output features Restricted to sklearn, we can check whether the transformer or estimator is a subclass of _OneToOneFeatureMixin , for example: from sklearn.decomposition import PCA from sklearn.preprocessing import StandardScaler from sklearn.feature_selection import SelectKBest from sklearn.base import _OneToOneFeatureMixin tf = {'pca':PCA(),'standardscaler':StandardScaler(),'kbest':SelectKBest()} [i+":"+str(issubclass(type(tf[i]),_OneToOneFeatureMixin)) for i in tf.keys()] ['pca:False', 'standardscaler:True', 'kbest:False']  These would the source code for _OneToOneFeatureMixin
Question: I'm unable to debug one error myself. I'm running python 3.8.12 inside docker image on Fedora release 35 (Thirty Five) and I'm unable to spawn threads from python. It's required for boto3 transfer to run in parallel and it uses concurrent.features to do so. The simplest example which replicates my issue without any dependencies is (copied from python docs) import concurrent.futures import urllib.request URLS = ['http://www.foxnews.com/', 'http://www.cnn.com/', 'http://europe.wsj.com/', 'http://www.bbc.co.uk/', 'http://some-made-up-domain.com/'] def load_url(url, timeout): with urllib.request.urlopen(url, timeout=timeout) as conn: return conn.read() with concurrent.futures.ThreadPoolExecutor(max_workers=5) as executor: future_to_url = {executor.submit(load_url, url, 60): url for url in URLS} for future in concurrent.futures.as_completed(future_to_url): url = future_to_url[future] try: data = future.result() except Exception as exc: pass  sadly output of these lines is Traceback (most recent call last): File "<stdin>", line 2, in <module> File "<stdin>", line 2, in <dictcomp> File "/usr/lib64/python3.8/concurrent/futures/thread.py", line 188, in submit self._adjust_thread_count() File "/usr/lib64/python3.8/concurrent/futures/thread.py", line 213, in _adjust_thread_count t.start() File "/usr/lib64/python3.8/threading.py", line 852, in start _start_new_thread(self._bootstrap, ()) RuntimeError: can't start new thread  That's all I have. Is there place where should I look? I've already checked ulimit which says unlimited. I'm kind of despair where to look or what to change to debug this issue.

Answer: Solution to this problem was to upgrade docker from version 18.06.1-ce to 20.10.7. Why?  This is because the default seccomp profile of Docker 20.10.9 is not adjusted to support the clone() syscall wrapper of glibc 2.34 adopted in Ubuntu 21.10 and Fedora 35.  Source: ubuntu:21.10 and fedora:35 do not work on the latest Docker (20.10.9)
Question: I am trying to convert a multiline json to a single line json. So the existing json file I have looks like this:  { "a": [ "b", "bill", "clown", "circus" ], "vers": 1.0 }  When I load the file, it comes in a dictionary and not sure how to strip blank spaces in the dictionary.  f = open('test.json') data = json.load(f)  What I would like it to come out is the following: {"a":["b","bill","clown","circus"],"vers":1} 

Answer: Using the standard library json you can get: import json with open('test.json') as handle: data = json.load(handle) text = json.dumps(data, separators=(',', ':')) print(text)  Result: {"a":["b","bill","clown","circus"],"vers":1.0}  Remark:  1.0 is not simplified to 1, probably, because this would change the type from float to int at Python level.
Question: I want to close all other windows opened by the main window when the main window is closed. Please find below the min. code that I was testing: from PyQt5.QtWidgets import QApplication, QMainWindow, QPushButton, QLabel, QVBoxLayout, QWidget import sys class AnotherWindow(QWidget): """ This "window" is a QWidget. If it has no parent, it will appear as a free-floating window as we want. """ def __init__(self): super().__init__() layout = QVBoxLayout() self.label = QLabel("Another Window") layout.addWidget(self.label) self.setLayout(layout) class MainWindow(QMainWindow): def __init__(self): super().__init__() self.button = QPushButton("Push for Window") self.button.clicked.connect(self.show_new_window) self.setCentralWidget(self.button) def show_new_window(self, checked): self.w = AnotherWindow() self.w.show() def close_another_window(self): if self.w: self.w.close() app = QApplication(sys.argv) w = MainWindow() app.aboutToQuit.connect(w.close_another_window) w.show() app.exec()  As shown above I tried using the aboutToQuit option of the QApplication, but it only gets called when the another window also is closed. I want to close the another window automaticaly when the mainwindow is closed.

Answer: Implement the closeEvent: class MainWindow(QMainWindow): w = None # ... def closeEvent(self, event): if self.w: self.w.close()  Note that you can also use QApplication.closeAllWindows() to close any top level window, even without having any direct reference, but if any of those windows ignores the closeEvent() the function will stop trying to close the remaining. To avoid that, you can cycle all windows using QApplication.topLevelWidgets(); windows ignoring the closeEvent will still keep themselves open, but all the others will be closed:  def closeEvent(self, event): for window in QApplication.topLevelWidgets(): window.close()
Question: We have a base model that sets a created and modified field: class BaseModel(models.Model): created = models.DateTimeField(_('created'), auto_now_add=True) modified = models.DateTimeField(_('modified'), auto_now=True) ... other default properties class Meta: abstract = True  We use this class to extend our models: class Event(BaseModel):  Is there a way to over ride the created date when creating new Events? This is a stripped down version of our code. We are sending an array of event objects containing a created timestamp in our request payload. After the objects are added to the db, the created property is set to now and not the value from the payload. I would like to still extend from the BaseModel as other areas of the code may not explicitly set a created value, in which case it should default to now. events = [] for e in payload['events']: event = Event( created=datetime.datetime.fromisoformat(e['created']) name='foo' ) events.append(event) Event.objects.bulk_create(events) 

Answer: You can override the created field for your Event model with: from django.utils.timezone import now class Event(BaseModel): created = models.DateTimeField(default=now) If you do not want this to show up for ModelForms and ModelAdmins by default, you can make use of editable=False [Django-doc]: from django.utils.timezone import now class Event(BaseModel): created = models.DateTimeField(default=now, editable=False)
Question: I have a simple Redis command that does the following: redis_conn.lpop(queue_name, batch_size)  According to the Redis documentation and their Python SDK documentation, this should be a valid request. And, yet, I get the following error:  redis.exceptions.ResponseError: wrong number of arguments for 'lpop' command  Maybe I'm being obtuse and making a noobie mistake because it's 2:00AM but, this should work. So why doesn't it?

Answer: Well, I was being obtuse. The documentation I linked states that the count argument is available from version 6.2. However, since I'm running Windows I don't get the newest version, ergo the failure.
Question: Assuming a sorted list of integers as below: data = [1] * 3 + [4] * 5 + [5] * 2 + [9] * 3 # [1, 1, 1, 4, 4, 4, 4, 4, 5, 5, 9, 9, 9]  I want to find the indices where the values changes, i.e. [3, 8, 10, 13]  One approach is to use itertools.groupby: cursor = 0 result = [] for key, group in groupby(data): cursor += sum(1 for _ in group) result.append(cursor) print(result)  Output [3, 8, 10, 13]  This approach is O(n). Another possible approach is to use bisect.bisect_left: cursor = 0 result = [] while cursor < len(data): cursor = bisect_left(data, data[cursor] + 1, cursor, len(data)) result.append(cursor) print(result)  Output [3, 8, 10, 13]  This approach is O(k*log n) where k is the number of distinct elements. A variant of this approach is to use an exponential search. Is there any faster or more performant way of doing this?

Answer: When it comes to asymptotic complexity I think you can improve slightly on the binary search on average when you apply a more evenly spread divide-and-conquer approach: try to first pinpoint the value-change that occurs closer to the middle of the input list, thereby splitting the range in approximately two halves, which would reduce the next binary search operation path by about one. Yet, because this is Python, the gain might not be noticeable, because of the Python-code overhead (like for yield, yield from, the recursion, ...). It might even perform worse for the list sizes you work with: from bisect import bisect_left def locate(data, start, end): if start >= end or data[start] == data[end - 1]: return mid = (start + end) // 2 val = data[mid] if val == data[start]: start = mid val += 1 i = bisect_left(data, val, start + 1, end) yield from locate(data, start, i) yield i yield from locate(data, i, end) data = [1] * 3 + [4] * 5 + [5] * 2 + [9] * 3 print(*locate(data, 0, len(data))) # 3 8 10  Note that this only outputs valid indices, so 13 is not included for this example input.
Question: I am working on a Python project where the Formatting is set to Autopep8. Is there any extension or possible settings where can define to exclude a certain line(s) from formatting when VS is set to format modified code/file on save or with keyboard shortcuts?

Answer: Add the comment # noqa to the end of each line you want VS Code to leave alone. For instance, to prevent VS Code from changing RED = 0 YELLOW = 1  to RED = 0 YELLOW = 1  just do the following: RED = 0 # noqa YELLOW = 1 # noqa
Question: I feel like this is a really simple question but I can't find the solution. Given a boolean array of true/false values, I need the output of all the indices with the value "false". I have a way to do this for true: test = [ True False True True] test1 = np.where(test)[0]  This returns [0,2,3], in other words the corresponding index for each true value. Now I need to just get the same thing for the false, where the output would be [1]. Anyone know how?

Answer: Use np.where(~test) instead of np.where(test).
Question: Spyder has a text wrap feature which includes a gray line at the end of the line. How do you remove it? 

Answer: On Spyder 5.0.0, You can change the line limit and remove the vertical bar by going to Settings -> Completion and linting -> Code style and formatting -> Line length -> Show vertical line at that length
Question: How can I save a file generated by colab notebook directly to github repo? It can be assumed that the notebook was opened from the github repo and can be (the same notebook) saved to the same github repo.

Answer: Google Colaboratory's integrating with github tends to be lacking, however you can run bash commands from inside the notebook. These allow you to access and modify any data generated. You'll need to generate a token on github to allow access to the repository you want to save data to. See here for how to create a personal access token. Once you have that token, you run git commands from inside the notebook to clone the repository, add whatever files you need to, and then upload them. This post here provides an overview of how to do it in depth. That being said, this approach is kind of cumbersome, and it might be preferable to configure colab to work over an SSH connection. Once you do that, you can mount a folder on the colab instance to a folder on your local machine using sshfs. This will allow you to access the colab as though it were any other folder on your machine, including opening it in your IDE, viewing files in a file browser, and cloning or updating git repositories. This goes more in depth on that. These are the best options I was able to identify, and I hope one of them can be made to work for you.
Question: I have a flask app that uses wtforms. I have a file which does: from wtforms.fields.html5 import DateField, EmailField, TelField # rest of the file  I just wanted to rebuild my docker container and now I have this error: ModuleNotFoundError: No module named 'wtforms.fields.html5'  I have in my requirements.txt: flask flask-login flask_sqlalchemy Flask-Mail pyodbc requests waitress wtforms  I tried to add flask_WTF but it did not fix it. Any idea what's going on? I thought of upgrading wtforms but it seems like I have the newest version: pip install wtforms Requirement already satisfied: wtforms in /usr/local/lib/python3.9/site-packages (3.0.0) Requirement already satisfied: MarkupSafe in /usr/local/lib/python3.9/site-packages (from wtforms) (2.0.1) 

Answer: Downgrading WTForms==2.3.3 solved the issue for me. Thread referenced here.
Question: Say you have 2 dataframes with the same columns. But say dataframe A has 10 rows, and dataframe B has 100 rows, but the 10 rows in dataframe A are in dataframe B. The 10 rows may not be in the same row numbers as dataframe B. How do we determine that those 10 rows in df A are fully contained in df B? For example. Say we have this for df A (only using 1 row) A | B | C 1 | 2 | 3  and df B is: A | B | C 2 | 5 | 5 3 | 2 | 7 1 | 2 | 3 5 | 1 | 5  How do we check that df A is contained in B? Assume that the rows will always be unique in the sense that there will always be a unique A+B combination

Answer: Is a Dataframe a subset of another: You can try solving this using merge and then comparison.  The inner-join of the 2 dataframes would be the same as the smaller dataframe if the second one is a superset for the first.  import pandas as pd # df1 - smaller dataframe, df2 - larger dataframe df1 = pd.DataFrame({'A ': [1], ' B ': [2], ' C': [3]}) df2 = pd.DataFrame({'A ': [2, 3, 1, 5], ' B ': [5, 2, 2, 1], ' C': [5, 7, 3, 5]}) df1.merge(df2).shape == df1.shape  True   If you have duplicates, then drop duplicates first - df1.merge(df2).drop_duplicates().shape == df1.drop_duplicates().shape  More details here.
Question: I want to extract how many positive reviews by brand are in a dataset which includes reviews from thousands of products. I used this code and I got a table including percentaje of positive and non-positive reviews. How can I get only the percentage of positive reviews by brand? I only want the "True" results in positive_review. Thanks! df_reviews_ok.groupby("brand")["positive_review"].value_counts(normalize=True).mul(100).round(2) brand positive_review Belkin False 70.00 True 30.00 Bowers & Wilkins False 67.65 True 32.35 Corsair False 75.22 True 24.78 Definitive Technology False 68.29 True 31.71 Dell False 60.87 True 39.13 DreamWave False 100.00 House of Marley False 100.00 JBL False 58.43 True 41.57 Kicker True 66.67 False 33.33 Lenovo False 76.92 True 23.08 Logitech False 75.75 True 24.25 MEE audio False 53.80 True 46.20 Microsoft False 67.86 True 32.14 Midland False 72.09 True 27.91 Motorola False 72.92 True 27.08 Netgear False 72.30 True 27.70 Pny False 68.78 True 31.22 Power Acoustik False 100.00 SVS False 100.00 Samsung False 61.94 True 38.06 Sanus False 75.93 True 24.07 Sdi Technologies, Inc. False 55.63 True 44.37 Siriusxm False 73.33 True 26.67 Sling Media False 67.16 True 32.84 Sony False 55.40 True 44.60 Toshiba False 56.52 True 43.48 Ultimate Ears False 70.21 True 29.79 Verizon Wireless False 75.86 True 24.14 WD False 58.33 True 41.67 Yamaha False 61.15 True 38.85 Name: positive_review, dtype: float64 

Answer: You can unstack the output and slice the True (df.groupby('brand') ['positive_review'].value_counts(normalize=True) .mul(100).round(2) .unstack(fill_value=0) [True] )