eshangj/stackoverflow_q_and_a_sample huggingface dataset:
Question: I am working fine with pandas_datareader, then today I installed below both yahoo finance from the below link trying to solve another issue. No data fetched Web.DataReader Panda pip install yfinance pip install fix_yahoo_finance  After the above installtion, pandas_datareader cannot be used anymore. I googled it and I did add the below import, and pandas_datareader is still not working. from urllib.parse import urlencode  Here is the error:  from pandas_datareader import data File "C:\Users\yongn\Anaconda3\lib\site-packages\pandas_datareader\__init__.py", line 2, in <module> from .data import ( File "C:\Users\yongn\Anaconda3\lib\site-packages\pandas_datareader\data.py", line 11, in <module> from pandas_datareader.av.forex import AVForexReader File "C:\Users\yongn\Anaconda3\lib\site-packages\pandas_datareader\av\__init__.py", line 6, in <module> from pandas_datareader.base import _BaseReader File "C:\Users\yongn\Anaconda3\lib\site-packages\pandas_datareader\base.py", line 7, in <module> from pandas.io.common import urlencode ImportError: cannot import name 'urlencode' 

Answer: I encountered exactly the same error. I am using python anaconda 2020_07 version. The solution is to use the latest pandas-datareader v0.9 from anaconda channel. If you use the pandas-datareader package from conda-forge which is using older version v0.8.1, you will encounter the error. This is the status as of 20Dec2020. I ran the command below to install the latest pandas-datareader package. conda install -c anaconda pandas-datareader The error message disappeared and the problem has been fixed. EDIT: If conda later downgrades pandas-datareader back to conda-forge older version, there's a fix. See https://stackoverflow.com/a/65386464/1709088
Question: foo.py: kwargs = {"a": 1, "b": "c"} def consume(*, a: int, b: str) -> None: pass consume(**kwargs)  mypy foo.py: error: Argument 1 to "consume" has incompatible type "**Dict[str, object]"; expected "int" error: Argument 1 to "consume" has incompatible type "**Dict[str, object]"; expected "str"  This is because object is a supertype of int and str, and is therefore inferred. If I declare: from typing import TypedDict class KWArgs(TypedDict): a: int b: str  and then annotate kwargs as KWArgs, the mypy check passes. This achieves type safety, but requires me to duplicate the keyword argument names and types for consume in KWArgs. Is there a way to generate this TypedDict from the function signature at type checking time, such that I can minimize the duplication in maintenance?

Answer: This will be available in Python 3.12 via PEP 692: from typing import TypedDict, Unpack, Required, NotRequired class KWArgs(TypedDict): a: Required[int] b: NotRequired[str] def consume(**kwargs: Unpack[KWArgs]) -> None: a = kwargs["a"] b = kwargs.get("b", ...) consume() # Not allowed. consume(a=1) # Allowed. consume(a=1, b="abc") # Allowed.
Question: In machine translation, we always need to slice out the first timestep (the SOS token) in the annotation and prediction. When using batch_first=False, slicing out the first timestep still keeps the tensor contiguous. import torch batch_size = 128 seq_len = 12 embedding = 50 # Making a dummy output that is `batch_first=False` batch_not_first = torch.randn((seq_len,batch_size,embedding)) batch_not_first = batch_first[1:].view(-1, embedding) # slicing out the first time step  However, if we use batch_first=True, after slicing, the tensor is no longer contiguous. We need to make it contiguous before we can do different operations such as view. batch_first = torch.randn((batch_size,seq_len,embedding)) batch_first[:,1:].view(-1, embedding) # slicing out the first time step output>>> """ --------------------------------------------------------------------------- RuntimeError Traceback (most recent call last) <ipython-input-8-a9bd590a1679> in <module> ----> 1 batch_first[:,1:].view(-1, embedding) # slicing out the first time step RuntimeError: view size is not compatible with input tensor's size and stride (at least one dimension spans across two contiguous subspaces). Use .reshape(...) instead. """  Does that mean batch_first=False is better, at least, in the context of machine translation? Since it saves us from doing the contiguous() step. Is there any cases that batch_first=True works better?

Answer: Performance There doesn't seem to be a considerable difference between batch_first=True and batch_first=False. Please see the script below: import time import torch def time_measure(batch_first: bool): torch.cuda.synchronize() layer = torch.nn.RNN(10, 20, batch_first=batch_first).cuda() if batch_first: inputs = torch.randn(100000, 7, 10).cuda() else: inputs = torch.randn(7, 100000, 10).cuda() torch.cuda.synchronize() start = time.perf_counter() for chunk in torch.chunk(inputs, 100000 // 64, dim=0 if batch_first else 1): _, last = layer(chunk) torch.cuda.synchronize() return time.perf_counter() - start print(f"Time taken for batch_first=False: {time_measure(False)}") print(f"Time taken for batch_first=True: {time_measure(True)}")  On my device (GTX 1050 Ti), PyTorch 1.6.0 and CUDA 11.0 here are the results: Time taken for batch_first=False: 0.3275816479999776 Time taken for batch_first=True: 0.3159054920001836  (and it varies either way so nothing conclusive). Code readability batch_first=True is simpler when you want to use other PyTorch layers which require batch as 0th dimension (which is the case for almost all torch.nn layers like torch.nn.Linear). In this case you would have to permute returned tensor anyway if batch_first=False was specified. Machine translation It should be better as the tensor is contiguous all the time and no copy of data has to be done. It also looks cleaner to slice using [1:] instead of [:,1:].
Question: I would like to know how to obtain the total number of CUDA Cores in my GPU using Python, Numba and cudatoolkit.

Answer: Most of what you need can be found by combining the information in this answer along with the information in this answer. We'll use the first answer to indicate how to get the device compute capability and also the number of streaming multiprocessors. We'll use the second answer (converted to python) to use the compute capability to get the "core" count per SM, then multiply that by the number of SMs. Here is a full example: $ cat t36.py from numba import cuda cc_cores_per_SM_dict = { (2,0) : 32, (2,1) : 48, (3,0) : 192, (3,5) : 192, (3,7) : 192, (5,0) : 128, (5,2) : 128, (6,0) : 64, (6,1) : 128, (7,0) : 64, (7,5) : 64, (8,0) : 64, (8,6) : 128, (8,9) : 128, (9,0) : 128 } # the above dictionary should result in a value of "None" if a cc match # is not found. The dictionary needs to be extended as new devices become # available, and currently does not account for all Jetson devices device = cuda.get_current_device() my_sms = getattr(device, 'MULTIPROCESSOR_COUNT') my_cc = device.compute_capability cores_per_sm = cc_cores_per_SM_dict.get(my_cc) total_cores = cores_per_sm*my_sms print("GPU compute capability: " , my_cc) print("GPU total number of SMs: " , my_sms) print("total cores: " , total_cores) $ python t36.py GPU compute capability: (5, 2) GPU total number of SMs: 8 total cores: 1024 $
Question: Do dict literals keep the order of their keys, in Python 3.7+? For example, is it guaranteed that {1: "one", 2: "two"} will always have its keys ordered this way (1, then 2) when iterating over it? (There is a thread in the Python mailing list with a similar subject, but it goes in all directions and I couldn't find an answer.) Similarly, is a dictionary like dict([('sape', 4139), ('guido', 4127), ('jack', 4098)]) ordered like in the list? The same question is true for other naturally ordered constructions, like dict comprehension and dict(sape=4139, guido=4127, jack=4098). PS: it is documented that dictionaries preserve insertion order. This question thus essentially asks: is it guaranteed that data is inserted in the order of a dict literal, of the list given to dict(), etc.

Answer: Yes, any method of constructing a dict preserves insertion order, in Python 3.7+.  For literal key-value pairs, see the documentation:  If a comma-separated sequence of key/datum pairs is given, they are evaluated from left to right to define the entries of the dictionary   See also: Martijn's answer on How to keep keys/values in same order as declared?  For comprehensions, from the same source:  When the comprehension is run, the resulting key and value elements are inserted in the new dictionary in the order they are produced.  Lastly, the dict initializer works by iterating over its argument and keyword arguments, and inserting each in order, similar to this: def __init__(self, mapping_or_iterable, **kwargs): if hasattr(mapping_or_iterable, "items"): # It's a mapping for k, v in mapping_or_iterable.items(): self[k] = v else: # It's an iterable of key-value pairs for k, v in mapping_or_iterable: self[k] = v for k, v in kwargs.items(): self[k] = v  [This is based on the source code, but glossing over a lot of unimportant details, e.g. that dict_init is just a wrapper on dict_update_common.] This, combined with the fact that keyword arguments pass a dictionary in the same order since Python 3.6, makes dict(x=…, y=…) preserve the order of the variables.
Question: Say I've got a function signature like this: def any_foo( bar: Bar, with_baz: Optional[Baz] = None, with_datetime: Optional[datetime] = None, effective: Optional[bool] = False, ) -> Foo  I could of course just copy its declaration and fiddle with it enough to create the following TypedDict: AnyFooParameters = TypedDict( "AnyFooParameters", { bar: Bar, with_baz: Optional[Baz], with_datetime: Optional[datetime], effective: Optional[bool] } )  But this seems like such a straight-forward transformation that I wonder whether there's some easy way to create this TypedDict (or at least the name: type pairs) straight from the function.

Answer: The result of any_foo.__annotations__ is exactly what you want. For example: from typing import Optional def any_foo( req_int: int, opt_float: Optional[float] = None, opt_str: Optional[str] = None, opt_bool: Optional[bool] = False, ) -> int: pass  And with any_foo.__annotations__, you can get this: {'req_int': int, 'opt_float': typing.Optional[float], 'opt_str': typing.Optional[str], 'opt_bool': typing.Optional[bool], 'return': int}  Note you can access to the type of return value by the return key. BTW, since return is a reserved keyword, you cannot name an argument as return so there's no need to warry about a duplicate key :)
Question: I'm getting started to AsyncIO and AioHTTP, and i'm writing some basic code to get familiar with the syntax. I tried the following code that should perform 3 requests concurrently: import time import logging import asyncio import aiohttp import json from aiohttp import ClientSession, ClientResponseError from aiocfscrape import CloudflareScraper async def nested(url): async with CloudflareScraper() as session: async with session.get(url) as resp: return await resp.text() async def main(): URL = "https://www.binance.com/api/v3/exchangeInfo" await asyncio.gather(nested(URL), nested(URL), nested(URL)) asyncio.run(main())  Here is the output: raise RuntimeError('Event loop is closed') RuntimeError: Event loop is closed  I don't understand why do i get that error, can anyone help me on this?

Answer: Update Originally I was recommending Greg's answer below: import asyncio import sys if sys.platform: asyncio.set_event_loop_policy(asyncio.WindowsSelectorEventLoopPolicy())  Turned out, using WindowsSelectorEventLoop has functionality issues such as:  Can't support more than 512 sockets Can't use pipe Can't use subprocesses  due to the fact that Windows uses I/O completion Ports unlike *nix - Therefore SelectorEventLoop is not designed for Windows nor is implemented as full. If those limitations matters to you - You might be better off using lengthy workaround in this answer. Check out more about differences at documents. Or alternatively, consider using Trio over asyncio, which is much more stable and consistent. import trio async def task(): await trio.sleep(5) trio.run(task)   Original post I've finally figured out how to keep ProactorEventLoop running, preventing unsuccessful IO closure. Really not sure why windows' Event loop is so faulty, as this also happens for asyncio.open_connection and asyncio.start_server. To workaround this, you need to run event loop in forever loop and close manually. Following code will cover both windows and other environments. import asyncio from aiocfscrape import CloudflareScraper async def nested(url): async with CloudflareScraper() as session: async with session.get(url) as resp: return await resp.text() async def main(): await nested("https://www.binance.com/api/v3/exchangeInfo") try: assert isinstance(loop := asyncio.new_event_loop(), asyncio.ProactorEventLoop) # No ProactorEventLoop is in asyncio on other OS, will raise AttributeError in that case. except (AssertionError, AttributeError): asyncio.run(main()) else: async def proactor_wrap(loop_: asyncio.ProactorEventLoop, fut: asyncio.coroutines): await fut loop_.stop() loop.create_task(proactor_wrap(loop, main())) loop.run_forever()  This code will check if new EventLoop is ProactorEventLoop. If so, keep loop forever until proactor_wrap awaits main and schedules loop stop. Else - possibly all other OS than Windows - doesn't need these additional steps, simply call asyncio.run() instead. IDE like Pycharm will complain about passing AbstractEventLoop to ProactorEventLoop parameter, safe to ignore.
Question: I am trying to update .env environment variables with python. With os.environ I am able to view and change local environment variables, but I want to change the .env file. Using python-dotenv I can load .env entries into local environment variables .env File key=value  test.py from dotenv import load_dotenv, find_dotenv load_dotenv(find_dotenv()) print(os.environ['key']) # outputs 'value' os.environ['key'] = "newvalue" print(os.environ['key']) # outputs 'newvalue'  .env File key=value  The .env File is not changed! Only the local env variable is changed. I could not find any documentation on how to update the .env file. Does anyone know a solution?

Answer: Use dotenv.set_key. import dotenv dotenv_file = dotenv.find_dotenv() dotenv.load_dotenv(dotenv_file) print(os.environ["key"]) # outputs "value" os.environ["key"] = "newvalue" print(os.environ['key']) # outputs 'newvalue' # Write changes to .env file. dotenv.set_key(dotenv_file, "key", os.environ["key"])
Question: In Python, What is the differences between module, sub-module, package and a sub-package?

Answer: package |-- __init__.py |-- module.py |-- sub_package |-- __init__.py |-- sub_module.py  Consider packages and sub-packages as folders and sub-folders containing init.py file with other python files. modules are the python files inside the package. sub-modules are the python files inside the sub-package.
Question: I am using a very small model for testing purposes using tensorflow 2.3 and keras. Looking at my terminal, I get the following warning: I tensorflow/compiler/mlir/mlir_graph_optimization_pass.cc:118] None of the MLIR optimization passes are enabled (registered 1)  However, the code works as expected. But what does this message mean? Thanks.

Answer: MLIR is being used as another solution to implementing and optimizing Tensorflow logic. This informative message is benign and is saying MLIR was not being used. This is expected as in TF 2.3, the MLIR based implementation is still being developed and proven, so end users are generally not expected to use the MLIR implementation and are instead expected to use the non-MLIR feature complete implementation. Update: still experimental on version 2.9.1. On the docs it is written:  DO NOT USE, DEV AND TESTING ONLY AT THE MOMENT.
Question: I try to create children of Foo, each should have its own uuid. In the real code no Instance of Foo will be created only it's children. The children will be saved in a database later, the uuid is to retrieve right objects from the database. In the first code snippet I tried to use the init method, which results in an AttributeError. I also tried to use a classmethod, which results in loosing all fields in my child objects. If I the second snippet every child gets the same uuid, which makes sense to me, as it's passed as default value. I could put the uuid creation into the children, though this feels wrong when using inheritance. Is there a better way to create a uuid for each child? # foo_init_.py class Foo(BaseModel): def __init__(self): self.id_ = uuid4() # >>> AttributeError: __fields_set__ # foo_classmethod.py class Foo(BaseModel): @classmethod def __init__(cls): cls.id_ = uuid4() # >>> Bar loses id_ fields  from uuid import uuid4, UUID from pydantic import BaseModel class Foo(BaseModel): id_: UUID = uuid4() class Bar(Foo): pass class Spam(Foo): pass if __name__ == '__main__': b1 = Bar() print(b1.id_) # >>> 73860f46-5606-4912-95d3-4abaa6e1fd2c b2 = Bar() print(b2.id_) # >>> 73860f46-5606-4912-95d3-4abaa6e1fd2c s1 = Spam() print(s1.id_) # >>> 73860f46-5606-4912-95d3-4abaa6e1fd2c 

Answer: You could use the default_factory parameter: class Foo(BaseModel): id_: UUID = Field(default_factory=uuid4)
Question: Some functions like numpy.intersect1d return differents types (in this case an ndarray or a tuple of three ndarrays) but the compiler can only infer one of them, so if I like to make: intersection: np.ndarray = np.intersect1d([1, 2, 3], [5, 6, 2])  It throws a type warning: Expected type 'ndarray', got 'Tuple[ndarray, ndarray, ndarray]' instead  I could avoid this kind of problems in other languages like Typescript where I could use the as keyword to assert the type (without impact in runtime). I've read the documentation and saw the cast function, but I'd to know if there is any inline solution or something I'm missing.

Answer: According to the MyPy documentation, there are two ways to do type assertions:  As an inline expression, you can use the typing.cast(..., ...) function. The docs say this is "usually" done to cast from a supertype to a subtype, but doesn't say you can't use it in other cases. As a statement, you can use assert isinstance(..., ...), but this will only work with concrete types like int or list which are represented at runtime, not more complex types like List[int] which can't be checked by isinstance.  Since the documentation doesn't mention any other ways to do type assertions, it seems like these are the only ways.
Question: A generator can be iterated step by step by using the next() built-in function. For example: def sync_gen(n): """Simple generator""" for i in range(n): yield i**2 sg = sync_gen(4) print(next(sg)) # -> 0 print(next(sg)) # -> 1 print(next(sg)) # -> 4  Using next() on an asynchronous generator does not work: import asyncio async def async_gen(n): for i in range(n): yield i**2 async def main(): print("Async for:") async for v in async_gen(4): # works as expected print(v) print("Async next:") ag = async_gen(4) v = await next(ag) # raises: TypeError: 'async_generator' object is not an iterator print(v) asyncio.run(main())  Does something like v = await async_next(ag) exist to obtain same behavior as with normal generators?

Answer: Since Python 3.10 there are aiter(async_iterable) and awaitable anext(async_iterator) builtin functions, analogous to iter and next, so you don't have to rely on the async_iterator.__anext__() magic method anymore. This piece of code works in python 3.10: import asyncio async def async_gen(n): for i in range(n): yield i**2 async def main(): print("Async next:") ag = async_gen(4) print(await anext(ag)) asyncio.run(main())
Question: pip install xmlsec commands throws the below error. ERROR: Command errored out with exit status 1: command: /home/xxx/PycharmProjects/saml_impl/saml_impl/venv/bin/python /home/sathia/PycharmProjects/saml_impl/saml_impl/venv/lib/python3.8/site-packages/pip/_vendor/pep517/_in_process.py build_wheel /tmp/tmpu_b5m5vz cwd: /tmp/pip-install-gblz98sr/xmlsec Complete output (14 lines): running bdist_wheel running build running build_py package init file 'src/xmlsec/__init__.py' not found (or not a regular file) creating build creating build/lib.linux-x86_64-3.8 creating build/lib.linux-x86_64-3.8/xmlsec copying src/xmlsec/py.typed -> build/lib.linux-x86_64-3.8/xmlsec copying src/xmlsec/template.pyi -> build/lib.linux-x86_64-3.8/xmlsec copying src/xmlsec/constants.pyi -> build/lib.linux-x86_64-3.8/xmlsec copying src/xmlsec/__init__.pyi -> build/lib.linux-x86_64-3.8/xmlsec copying src/xmlsec/tree.pyi -> build/lib.linux-x86_64-3.8/xmlsec running build_ext error: Unable to invoke pkg-config. ---------------------------------------- ERROR: Failed building wheel for xmlsec Failed to build xmlsec ERROR: Could not build wheels for xmlsec which use PEP 517 and cannot be installed directly'  I don't know how to resolve this issue. I tried to install other xmlsec package too, nothing worked.

Answer: Xmlsec listed here https://pypi.org/project/xmlsec/. The below command should install for download required native libraries. sudo apt-get install pkg-config libxml2-dev libxmlsec1-dev libxmlsec1-openssl
Question: I'm using bandit to check my code for potential security issues: bandit -r git-repository/  However, the most common item found by bandit is B101. It is triggered by assert statements within tests. I use pytest, so this is not a concern, but a good practice. I've now created a .bandit file with [bandit] skips: B101  But that also skips a lot of other code. Is there a solution to this issue?

Answer: A possible solution is to tell bandit to skip tests altogether. Assuming your code lives in a src subfolder, run bandit --configfile bandit.yaml --recursive src  with the following bandit.yaml in the project's root directory # Do not check paths including `/tests/`: # they use `assert`, leading to B101 false positives. exclude_dirs: - '/tests/'  There is a bunch of related issues and pull requests. Update: I like Diego's solution better.
Question: Is there a way to find and replace text for Jupyter Notebooks in Visual Studio Code. I can do it for a specific cell by clicking to that cell and pressing Ctrl+H. But I cannot find a way to do it for all the cells in the entire notebook. This is how it looks like when I press Ctrl+H for specific cells: 

Answer: This issue no longer exists in Visual Studio Code as of Version 1.59.1. You can use Ctrl + H to find/replace in the whole Jupyter Notebook.
Question: For a plotly figure factory distribution plot, the default distribution is kde (kernel density estimation):  You can override the default by setting curve = 'normal' to get:  But how can you show both kde and the normal curve in the same plot? Assigning a list like curve_type = ['kde', 'normal'] will not work. Complete code: import plotly.figure_factory as ff import plotly.graph_objects as go import plotly.express as px import numpy as np np.random.seed(2) x = np.random.randn(1000) hist_data = [x] group_labels = ['distplot'] # name of the dataset mean = np.mean(x) stdev_pluss = np.std(x) stdev_minus = np.std(x)*-1 fig = ff.create_distplot(hist_data, group_labels, curve_type='kde') fig.update_layout(template = 'plotly_dark') fig.show() 

Answer: The easiest thing to do is build another figure fig2 with curve_type = 'normal' and pick up the values from there using: fig2 = ff.create_distplot(hist_data, group_labels, curve_type = 'normal') normal_x = fig2.data[1]['x'] normal_y = fig2.data[1]['y']  And then inlclude those values in the first fig using fid.add_trace(go.Scatter()) like this: fig2 = ff.create_distplot(hist_data, group_labels, curve_type = 'normal') normal_x = fig2.data[1]['x'] normal_y = fig2.data[1]['y'] fig.add_traces(go.Scatter(x=normal_x, y=normal_y, mode = 'lines', line = dict(color='rgba(0,255,0, 0.6)', #dash = 'dash' width = 1), name = 'normal' )) fig.show()  Plot with two density curves: