eshangj/stackoverflow_q_and_a_sample huggingface dataset:
Question: I have a script with the following import: from datetime import datetime  and a piece of code where I call: datetime.fromisoformat(duedate)  Sadly, when I run the script with an instance of Python 3.6, the console returns the following error:  AttributeError: type object 'datetime.datetime' has no attribute 'fromisoformat'  I tried to run it from two instances of anaconda (3.7 and 3.8) and it works nice and smooth. I supposed there was an import problem so I tried to copy datetime.py from anaconda/Lib to the script directory, with no success. The datetime.py clearly contains the class datetime and the method fromisoformat but still it seems unlinked. I even tried to explicitly link the datetime.py file, with the same error: parent_dir = os.path.abspath(os.path.dirname(__file__)) vendor_dir = os.path.join(parent_dir, 'libs') sys.path.append(vendor_dir+os.path.sep+"datetime.py")  Can you help me? My ideas are over...

Answer: The issue here is actually that fromisoformat is not available in Python versions older than 3.7, you can see that clearly stated in the documenation here. Return a date corresponding to a date_string given in the format YYYY-MM-DD: >>> >>> from datetime import date >>> date.fromisoformat('2019-12-04') datetime.date(2019, 12, 4) This is the inverse of date.isoformat(). It only supports the format YYYY-MM-DD. New in version 3.7.
Question: I have a dataframe: df = col1 col2 col3 1 2 3 1 4 6 3 7 2  I want to edit df, such that when the value of col1 is smaller than 2 , take the value from col3. So I will get: new_df = col1 col2 col3 3 2 3 6 4 6 3 7 2  I tried to use assign and df.loc but it didn't work. What is the best way to do so?

Answer: df['col1'] = df.apply(lambda x: x['col3'] if x['col1'] < x['col2'] else x['col1'], axis=1)
Question: I'm looking for a conceptual answer on this question. I'm wondering whether using ThreadPool in python to perform concurrent tasks, guarantees that data is not corrupted; I mean multiple threads don't access the critical data at the same time. If so, how does this ThreadPoolExecutor internally works to ensure that critical data is accessed by only one thread at a time?

Answer: Thread pools do not guarantee that shared data is not corrupted. Threads can swap at any byte code execution boundary and corruption is always a risk. Shared data should be protected by synchronization resources such as locks, condition variables and events. See the threading module docs concurrent.futures.ThreadPoolExecutor is a thread pool specialized to the concurrent.futures async task model. But all of the risks of traditional threading are still there.  If you are using the python async model, things that fiddle with shared data should be dispatched on the main thread. The thread pool should be used for autonomous events, especially those that wait on blocking I/O.
Question: I am trying to set the navigator.webdriver variable in the Firefox browser to undefined using Selenium in Python. I have been able to successfully do this when using Chrome, but now I need to do the same thing using in Firefox. When using the Firefox webdriver, execute_cdp_cmd(...) does not exist. Does anyone know how to do the same thing using the firefox webdriver instead of the chrome webdriver? Please see the relevant code below. driver.execute_cdp_cmd("Page.addScriptToEvaluateOnNewDocument", { "source": """ Object.defineProperty(navigator, 'webdriver', { get: () => undefined }) """ }) 

Answer: I have since found a solution to my problem. The code below will set "navigator.webdriver" to undefined in a Firefox browser being run by Selenium. profile.set_preference("dom.webdriver.enabled", False)
Question: I'm trying to record screen when playing a website by using mss and opencv, but I don't want the program to use the current screen. I want to put them to play on a second desktop, like Desktop 2 in the following picture macos have 4 desktop setup so I can work in the desktop 1 without any interruption.

Answer: Currently, MSS does not support capturing inactive workspaces. This Ask Ubuntu answer indicates (referencing a post on an "archived" (i.e. apparently deleted) forum) that this is not normally possible in the X Window System.1 The answer discusses using Xvfb as a workaround, but this does not appear to be useful for screen-capture software, as it is essentially a way to run the application on a virtual display, which can then be captured normally. If taking a screenshot of an inactive space is possible on macOS (which I consider unlikely for the same reasons as on X), you would likely need to use non-API functions from CoreGraphics (as there was no public API for spaces as of 2016). This GitHub repository documents those functions, though the repository was last updated in 2016, so it may not be as helpful as you would like. Another option, which may be the least OS-dependent, is to run a headless virtual machine and take screenshots of that. How well this works will depend on the virtual machine manager and the virtual machine itself, as well as how you are taking the screenshots. 1 Basically because the inactive desktops are not rendered.  For anyone who sees this question while looking for a way to take screenshots of other monitors, look at the edit history of this answer.
Question: I created a Gensim LDA Model as shown in this tutorial: https://www.machinelearningplus.com/nlp/topic-modeling-gensim-python/ lda_model = gensim.models.LdaMulticore(data_df['bow_corpus'], num_topics=10, id2word=dictionary, random_state=100, chunksize=100, passes=10, per_word_topics=True)  And it generates 10 topics with a log_perplexity of:   lda_model.log_perplexity(data_df['bow_corpus']) = -5.325966117835991  But when I run the coherence model on it to calculate coherence score, like so: coherence_model_lda = CoherenceModel(model=lda_model, texts=data_df['bow_corpus'].tolist(), dictionary=dictionary, coherence='c_v') with np.errstate(invalid='ignore'): lda_score = coherence_model_lda.get_coherence()  My LDA-Score is nan. What am I doing wrong here?

Answer: Solved! Coherence Model requires the original text, instead of the training corpus fed to LDA_Model - so when i ran this: coherence_model_lda = CoherenceModel(model=lda_model, texts=data_df['corpus'].tolist(), dictionary=dictionary, coherence='c_v') with np.errstate(invalid='ignore'): lda_score = coherence_model_lda.get_coherence()   I got a coherence score of: 0.462  Hope this helps someone else making the same mistake. Thanks!
Question: Using the vanilla configuration of base BERT model in the huggingface implementation, I get a tuple of length 2. import torch import transformers from transformers import AutoModel,AutoTokenizer bert_name="bert-base-uncased" tokenizer = AutoTokenizer.from_pretrained(bert_name) BERT = AutoModel.from_pretrained(bert_name) e=tokenizer.encode('I am hoping for the best', add_special_tokens=True) q=BERT(torch.tensor([e])) print (len(q)) #Output: 2  The first element is what I expect to receive - the 768 dimension embedding of each input token. print (e) #Output : [101, 1045, 2572, 5327, 2005, 1996, 2190, 102] print (q[0].shape) #Output : torch.Size([1, 8, 768])  But what is the second element in the tuple? print (q[1].shape) # torch.Size([1, 768])  It has the same size as the encoding of each token. But what is it? Maybe a copy of the [CLS] token, a representation for the classification of the entire encoded text? Let's check. a= q[0][:,0,:] b=q[1] print (torch.eq(a,b)) #Output : Tensor([[False, False, False, .... False]])  Nope! What about a copy the embedding of the last token (for whatever reason)? c= q[0][:,-1,:] b=q[1] print (torch.eq(a,c)) #Output : Tensor([[False, False, False, .... False]])  So, also not that. The documentation talks about how changing the config can result in more tuple elements (like hidden states), but I did not find any description of this "mysterious" tuple element outputted by the default configuration. Any ideas as to what is it and what is its usage? 

Answer: The output in this case is a tuple of (last_hidden_state, pooler_output). You can find documentation about what the returns could be here.
Question: In the code below I created the net_plan_dict variable dictionary and converted it into a YAML format file. Inside the dictionary I have a field called addresses which is an array of three elements. After creating the YAML file, the three array elements were not placed under the addresses field : import yaml net_plan_dict = { 'networking': { 'addresses': ['192.168.1.1', '192.168.1.2', "192.168.1.3"], 'gateway4': '192.168.121.1' } } with open("new.yaml", "w") as f: yaml.dump(net_plan_dict, f)  Output of the above code is as follows (in the file below, the IPs are not below the address and have no space or indent). new.yaml: networking: addresses: - 192.168.1.1 <-------- does not have indent - 192.168.1.2 - 192.168.1.3 gateway4: 192.168.121.1  but my goal is to get this output file (how to create this file, when ips are under addresses field): networking: addresses: - 192.168.1.1 - 192.168.1.2 - 192.168.1.3 gateway4: 192.168.121.1 

Answer: PyYAML's dump() doesn't have the fine control to have a different indent for the mappings (2 positions) and sequences (4 positions), nor can it offset the sequence indicator (-) within the space of the (sequence) indent). If you want that kind of control over your output you should use ruamel.yaml (disclaimer: I am the author of that package): import sys import ruamel.yaml net_plan_dict = { 'networking': { 'addresses': ['192.168.1.1', '192.168.1.2', "192.168.1.3"], 'gateway4': '192.168.121.1' } } yaml = ruamel.yaml.YAML() yaml.indent(mapping=2, sequence=4, offset=2) yaml.dump(net_plan_dict, sys.stdout)  which gives: networking: addresses: - 192.168.1.1 - 192.168.1.2 - 192.168.1.3 gateway4: 192.168.121.1
Question: Let's say I have an async generator like this: async def event_publisher(connection, queue): while True: if not await connection.is_disconnected(): event = await queue.get() yield event else: return  I consume it like this: published_events = event_publisher(connection, queue) async for event in published_events: # do event processing here  It works just fine, however when the connection is disconnected and there is no new event published the async for will just wait forever, so ideally I would like to close the generator forcefully like this: if connection.is_disconnected(): await published_events.aclose()  But I get the following error: RuntimeError: aclose(): asynchronous generator is already running Is there a way to stop processing of an already running generator?

Answer: It seems to be related to this issue. Noticable:  As shown in https://gist.github.com/1st1/d9860cbf6fe2e5d243e695809aea674c, it's an error to close a synchronous generator while it is being iterated. ... In 3.8, calling "aclose()" can crash with a RuntimeError. It's no longer possible to reliably cancel a running asynchrounous generator.  Well, since we can't cancel running asynchrounous generator, let's try to cancel its running. import asyncio from contextlib import suppress async def cancel_gen(agen): task = asyncio.create_task(agen.__anext__()) task.cancel() with suppress(asyncio.CancelledError): await task await agen.aclose() # probably a good idea, # but if you'll be getting errors, try to comment this line  ... if connection.is_disconnected(): await cancel_gen(published_events)  Can't test if it'll work since you didn't provide reproducable example.
Question: I have a test that makes sure a specific (helpful) error message is raised, when a required package is not available. def foo(caller): try: import pkg except ImportError: raise ImportError(f'Install "pkg" to use {caller}') pkg.bar() with pytest.raises(ImportError, match='Install "pkg" to use test_function'): foo('test_function')  However, pkg is generally available, as other tests rely on it. Currently, I set up an additional virtual env without pkg just for this test. This seems like overkill. Is it possible to "hide" an installed package within a module or function?

Answer: I ended up with the following pytest-only solution, which appears to be more robust in the setting of a larger project. import builtins import pytest @pytest.fixture def hide_available_pkg(monkeypatch): import_orig = builtins.__import__ def mocked_import(name, *args, **kwargs): if name == 'pkg': raise ImportError() return import_orig(name, *args, **kwargs) monkeypatch.setattr(builtins, '__import__', mocked_import) @pytest.mark.usefixtures('hide_available_pkg') def test_message(): with pytest.raises(ImportError, match='Install "pkg" to use test_function'): foo('test_function')
Question: Suppose I have two lists: a = ['30', '10', '90', '1111', '17'] b = ['60', '1201', '30', '17', '900']  How would you sort this most efficiently, such that: list b is sorted with respect to a. Unique elements in b should be placed at the end of the sorted list. Unique elements in a can be ignored. example output: c = ['30', '17', '60', '1201', '900']  Sorry, it's a simple question. My attempt is stuck at the point of taking the intersection.  intersection = sorted(set(a) & set(b), key = a.index) 

Answer: There is no need to actually sort here. You want the elements in a which are in b, in the same order as they were in a; followed by the elements in b which are not in a, in the same order as they were in b. We can just do this with two filters, using the sets for fast membership tests: >>> a = ['30', '10', '90', '1111', '17'] >>> b = ['60', '1201', '30', '17', '900'] >>> a_set = set(a) >>> b_set = set(b) >>> [*filter(lambda x: x in b_set, a), *filter(lambda x: x not in a_set, b)] ['30', '17', '60', '1201', '900']  Or if you prefer comprehensions: >>> [*(x for x in a if x in b_set), *(x for x in b if x not in a_set)] ['30', '17', '60', '1201', '900']  Both take linear time, which is better than sorting.
Question: The most common way to patch something in a module seems to be to use something like from unittest.mock import patch from mypackage.my_module.my_submodule import function_to_test @patch('mypackage.my_module.my_submodule.fits.open') def test_something(self, mock_fits_open) # ... mock_fits_open.return_value = some_return_value function_to_test() # ...  However, with the value passed to the patch decorator being a string, I don't get lots of the nice benefits from IDE. I can't use parts of the string to jump to definitions. I don't get autocomplete (and an implicit spelling check). Nor full refactoring capabilities. And so on. Using patch.object I can get much closer to what I'm looking for. from unittest.mock import patch import mypackage.my_module.my_submodule from mypackage.my_module.my_submodule import function_to_test @patch.object(mypackage.my_module.my_submodule.fits, 'open') def test_something(self, mock_fits_open) # ... mock_fits_open.return_value = some_return_value function_to_test() # ...  However, this still requires the final part of the name of the referenced object is just a string. Is there a (nice) way to patch an object purely on the reference to that object? That is, I would like to be able to do something like from unittest.mock import patch import mypackage.my_module.my_submodule from mypackage.my_module.my_submodule import function_to_test @patch.reference(mypackage.my_module.my_submodule.fits.open) def test_something(self, mock_fits_open) # ... mock_fits_open.return_value = some_return_value function_to_test() # ... 

Answer: Patching works by replacing in the namespace where the name is looked up. The underlying logic of mock.patch is essentially working with a context-managed name shadowing. You could do the same thing manually with:  save original value associated with name (if any) try overwriting the name execute the code under test finally resetting name back to the original value  Therefore, you fundamentally need to patch on a name, there is no patching a reference directly.
Question: tl;dr Environment variables set in a zappa_settings.json don't upload as environment variables to AWS Lambda. Where do they go? ts;wm I have a Lambda function configured, deployed and managed using the Zappa framework. In the zappa_settings.json I have set a number of environment variables. These variables are definitely present as my application successfully runs however, when trying to inspect the Lambda function environment variables in the console or AWS CLI I see no environment variables have been uploaded to the Lambda function itself. Extract from zappa_settings.json: { "stage-dev": { "app_function": "project.app", "project_name": "my-project", "runtime": "python3.7", "s3_bucket": "my-project-zappa", "slim_handler": true, "environment_variables": { "SECRET": "mysecretvalue" } } }  Output of aws lambda get-function-configuration --function-name my-project-stage-dev: { "Configuration": { "FunctionName": "my-project-stage-dev", "FunctionArn": "arn:aws:lambda:eu-west-1:000000000000:function:my-project-stage-dev", "Runtime": "python3.7", "Role": "arn:aws:iam::000000000000:role/lambda-execution-role", "Handler": "handler.lambda_handler", "CodeSize": 12333025, "Description": "Zappa Deployment", "Timeout": 30, "MemorySize": 512, "LastModified": "...", "CodeSha256": "...", "Version": "$LATEST", "TracingConfig": { "Mode": "PassThrough" }, "RevisionId": "..." }, "Code": { "RepositoryType": "S3", "Location": "..." } }  Environment is absent from the output despite being included in the zappa_settings and the AWS documentation indicating it should be included if present, this is confirmed by checking in the console. I want to know where zappa is uploading the environment variables to, and if possible why it is doing so over using Lambda's in-built environment? AWS CLI docs: https://docs.aws.amazon.com/cli/latest/reference/lambda/get-function-configuration.html

Answer: environment_variables are saved in zappa_settings.py when creating a package for deployment (run zappa package STAGE and explore the archive) and are then dynamically set as environment variables by modifying os.environ in handler.py. To set native AWS variables you need to use aws_environment_variables.
Question: Suppose I have function that takes type as argument and returns instance of that type: def fun(t): return t(42)  Then I can call it and get objects of provided types: fun(int) # 42 fun(float) # 42.0 fun(complex) # (42+0j) fun(str) # "42" fun(MyCustomType) # something  That list is not exhaustive, I'd like to be able to use any type with appropriate constructor. Then, I'd like to add type hints for that function. What should be the type hint for return value of that function?  I've tried using simply t, as t is a type: def fun(t: type) -> t: return t(42)  but that doesn't work:  main.py:1: error: Name 't' is not defined   This answer suggests using a TypeVar: from typing import TypeVar T = TypeVar("T") def fun(t: T) -> T: return t(42)  But that doesn't seem to be right, as T denotes a type, so it suggests that type itself is returned, not its instance. Mypy rejects it:  main.py:6: error: "object" not callable   Using Any obviously work, but I feel it's too vague, it doesn't convey the intent: from typing import Any def fun(t: type) -> Any: return t(42) 

Answer: TLDR: You need a TypeVar for the return type of calling t: def fun(t: Callable[[int], R]) -> R: ...   Constraining on a type is too restrictive here. The function accepts any Callable that takes an integer, and the return type of the function is that of the Callable. This can be specified using a TypeVar for the return type: from typing import Callable, TypeVar R = TypeVar('R') # the variable return type def fun(t: Callable[[int], R]) -> R: return t(42) fun(int) # Revealed type is 'builtins.int*' fun(float) # Revealed type is 'builtins.float*' reveal_type(fun(lambda x: str(x))) # Revealed type is 'builtins.str*'  This works for types as well, because type instantiation is a call. If a more complex signature, e.g. with keyword arguments, is needed, use Protocol (from typing or typing_extensions).  Note that if one explicitly wants to pass only 42 to the Callable, Literal (from typing or typing_extensions) can be used to specify that. R = TypeVar('R') def fun(t: Callable[[Literal[42]], R]) -> R: return t(42)  Note that any function of the type Callable[[int], R] also satisfies Callable[[Literal[42]], R].
Question: I am trying to use floodFill on an image like below to extract the sky:  But even when I set the loDiff=Scalar(0,0,0) and upDiff=Scalar(255,255,255) the result is just showing the seed point and does not grow larger (the green dot):  code: Mat flood; Point seed = Point(180, 80); flood = imread("D:/Project/data/1.jpeg"); cv::floodFill(flood, seed, Scalar(0, 0, 255), NULL, Scalar(0, 0, 0), Scalar(255, 255, 255)); circle(flood, seed, 2, Scalar(0, 255, 0), CV_FILLED, CV_AA);  This is the result (red dot is the seed):  How can I set the function to get a larger area (like the whole sky)?

Answer: You need to set loDiff and upDiff arguments correctly.  See floodFill documentation:   loDiff – Maximal lower brightness/color difference between the currently observed pixel and one of its neighbors belonging to the component, or a seed pixel being added to the component. upDiff – Maximal upper brightness/color difference between the currently observed pixel and one of its neighbors belonging to the component, or a seed pixel being added to the component.  Here is a Python code sample:  import cv2 flood = cv2.imread("1.jpeg"); seed = (180, 80) cv2.floodFill(flood, None, seedPoint=seed, newVal=(0, 0, 255), loDiff=(5, 5, 5, 5), upDiff=(5, 5, 5, 5)) cv2.circle(flood, seed, 2, (0, 255, 0), cv2.FILLED, cv2.LINE_AA); cv2.imshow('flood', flood) cv2.waitKey(0) cv2.destroyAllWindows()   Result:
Question: I feel Black is doing something not compliant (with my Organisation), so I am trying to ignore certain rules.  Example below and a related link PEP 8: whitespace before ':' My Organisation (Coding Standards) does not give priority to what Black feels is right, but wants a way to customise black configurations. I dont see any mentioned of Ignoring a Rule in Black documentation https://github.com/psf/black#command-line-options.  They have given examples to ignore Flake8 rules, but dont seem to have any documentation for their own product.

Answer: You can't customize black. From the readme:  Black reformats entire files in place. It is not configurable.
Question: currently I am using marshmallow schema to validate the request, and I have this a list and I need to validate the content of it. class PostValidationSchema(Schema): checks = fields.List( fields.String(required=True) )  the checks field is a list it should only contain these specific values ["booking", "reservation", "flight"]

Answer: If you mean to check the list only has those three elements in that order, then use Equal validator. from marshmallow import Schema, fields, validate class PostValidationSchema(Schema): checks = fields.List( fields.String(required=True), validate=validate.Equal(["booking", "reservation", "flight"]) ) schema = PostValidationSchema() schema.load({"checks": ["booking", "reservation", "flight"]}) # OK schema.load({"checks": ["booking", "reservation"]}) # ValidationError  If the list can have any number of elements and those can only be one of those three specific values, then use OneOf validator. from marshmallow import Schema, fields, validate class PostValidationSchema(Schema): checks = fields.List( fields.String( required=True, validate=validate.OneOf(["booking", "reservation", "flight"]) ), ) schema = PostValidationSchema() schema.load({"checks": ["booking", "reservation", "flight"]}) # OK schema.load({"checks": ["booking", "reservation"]}) # OK schema.load({"checks": ["booking", "dummy"]}) # ValidationError