eshangj/stackoverflow_q_and_a_sample huggingface dataset:
Question: Here is my function: def call_func(func, *args): return func(*args)  I think I have two options here:  Using TypeVarTuple -> in Callable[[*Ts], Any] form. Ts = TypeVarTuple("Ts") T = TypeVar("T") def call_func(func: Callable[[*Ts], T], *args: *Ts) -> T: return func(*args)  Currently Mypy has problem with [*Ts] part. it says: Invalid type comment or annotation. (I also enabled --enable-incomplete-feature=TypeVarTuple.)  Using ParamSpec -> in Callable[P, Any] form. P = ParamSpec("P") T = TypeVar("T") def call_func(func: Callable[P, T], *args: P.args) -> T: return func(*args)  This time Mypy says: ParamSpec must have "*args" typed as "P.args" and "**kwargs" typed as "P.kwargs". It looks like it wants me to also specify kwargs.   What is the correct way of doing it? Is there any technical difference between using TypeVarTuple and ParamSec in Callable?

Answer: The only way to fully type-check something that only accepts positional-only arguments is with PEP 646, which mypy (as of the time of this answer) does not fully implement. However, you can manipulate typing.ParamSpec and typing.Protocol to make positional-only and keyword-only callable types which throw errors at a call site if someone tries to pass keyword and positional arguments, respectively. The core idea is to form a union with Callable[P, R] with some Protocol CallbackProto such that:  Callable[P, R] unioned with CallbackProto::__call__(self, *args: typing.Any) -> R rejects all attempts to pass any keyword arguments; Callable[P, R] unioned with CallbackProto::__call__(self, **kwargs: typing.Any) -> R rejects all attempts to pass any positional arguments.   In your case, let's say that this was the desired output: P = ParamSpec("P") R = TypeVar("R") def call_func(func: Callable[P, R], *args: P.args, **kwargs: P.kwargs) -> R: return func(*args) >>> def function_(a: int, b: str) -> None: ... ... ... >>> call_func(function_, 1, "") # OK >>> call_func(function_, 1, b="") # Unexpected keyword argument "b"  This can be done by making call_func reject all attempts to pass keyword arguments: from __future__ import annotations import typing as t if t.TYPE_CHECKING: import collections.abc as cx P = t.ParamSpec("P") R = t.TypeVar("R") CallFuncT = t.TypeVar("CallFuncT", bound="CallFunc") class PositionalOnlyCallable(t.Protocol): def __call__(self, *args: t.Any) -> t.Any: ... class CallFunc(t.Protocol): def __call__(self, func: cx.Callable[P, R], *args: P.args, **kwargs: P.kwargs) -> R: ... def asPositionalOnlyCallable(f: CallFuncT, /) -> CallFuncT | PositionalOnlyCallable: """ No-op decorator which manipulates the typing signature. Unions a given callable's type with something which only accepts positional arguments. """ return f @asPositionalOnlyCallable def call_func(func: cx.Callable[P, R], *args: P.args, **kwargs: P.kwargs) -> R: return func(*args)  >>> def function_(a: int, b: str) -> None: ... ... ... >>> call_func(function_, 1, "") # OK >>> call_func(function_, 1, b="") # mypy: Unexpected keyword argument "b" for "__call__" of "PositionalOnlyCallable" [call-arg]
Question: i am trying to make an automation project which will use proxy to register in a site. but i am unable to get it work. I have tried selenium-wire for proxy authentication still it doesn't work from seleniumwire import webdriver from selenium.webdriver.common.by import By from selenium.webdriver.firefox.options import Options import os import keyboard import pyautogui from time import sleep user = 'jesbeqki' pwd = '1wbt8dnqtu8w' end = '2.56.119.93:5074' seleniumwire_options ={ "proxies" : { "https": "http://"f"{user}:{pwd}@{end}/", "http": "http://"f"{user}:{pwd}@{end}/", 'verify_ssl': False } } firefox_options = Options() # firefox_options.add_argument("--headless") driver = webdriver.Firefox(options=firefox_options, seleniumwire_options=seleniumwire_options) driver.get('https://httpbin.org/ip') text = driver.find_element(By.TAG_NAME, 'body').text print(text)  it's showing my original ip.

Answer: You can use SeleniumBase to proxy to a site. pip install seleniumbase, then run this with python after filling in the proxy details: from seleniumbase import SB with SB(uc=True, proxy="USER:PASS@IP:PORT") as sb: sb.driver.get("https://whatismyip.com") sb.sleep(5)
Question: When I'm running pytest normally it works perfectly fine (before first test it's upgrading migration and after last test it's downgrading migration). Without using xdist I can just set fixture scope='session' and its work that way. But when I run pytest-xdist (tests in parallel) it's upgrading migration before each test and downgrading migration after each test (I want to setUp before first test - first worker, and tearDown after last test - when last worker finishes). I think that setUp before first worker I can handle with File Lock (and also I don't need it that much because in migration upgrade I'm just creating some tables if not exist so if it was already upgraded it just won't do anything). I'm interested especially in tearDown after last worker finshes. import pytest import os import subprocess @pytest.fixture(autouse=True, scope='session') def handle_database(): """Create necesarry tables in db (using migration) and drop them after tests""" # Change directory to test/api_simulator os.chdir('test/api_simulator') # Perform migration upgrade subprocess.run(['flask', 'db', 'upgrade']) print('\n\nUpgrade migration\n\n') # Change directory back to the original directory os.chdir('../..') yield # Change directory to test/api_simulator os.chdir('test/api_simulator') # Perform migration downgrade subprocess.run(['flask', 'db', 'downgrade']) print('\n\nDowngrade migration\n\n') # Change directory back to the original directory os.chdir('../..')  I was searching about this issue in web but I found nothing good about tearDown with xdist.

Answer: The fixture is running as part of a test, even if it's only once in session scope. This means it's running in the process xdist opened. To run this only once you move the code to pytest_sessionstart and pytest_sessionfinish in conftest.py. Those methods will be executed once in "regular" scope before/after xdist kicks in. Note that the pytest_session functions will run again as part of the tests cycle so you need to check it def __handle_database(config, command): if not hasattr(config, 'workerinput'): os.chdir('test/api_simulator') # Perform migration upgrade subprocess.run(['flask', 'db', command]) print('\n\nUpgrade migration\n\n') # Change directory back to the original directory os.chdir('../..') def pytest_sessionstart(session): __handle_database(session.config, 'upgrade') def pytest_sessionfinish(session, exitstatus): __handle_database(session.config, 'downgrade')
Question: I've been working with Jupyter Notebooks for quite a while. When working with visualisations, I like to copy the output image from a cell by right clicking the image and selecting "Copy Image" from the context menu:  I like working with the direct copy from the notebook, especially for answering questions on Stack Overflow, so I'd rather not store them to disk. That would be a real reason to revert to legacy notebooks for me. However with the Notebook 7 migration coming, I gave the beta a try by running pip install notebook --pre --upgrade and to my surprise I can't right click, copy the output image because the new Jupyter context menu pops up instead.  This really breaks my workflow. How can I copy an image from the output of a cell in notebook 7+?

Answer: Hold down Shift while right-clicking to access the native browser menu, as spelled out here. For context, Version 7 of Jupyter Notebook is built off JupyterLab components (see here and here), and that means that some of the interface and GUI abilities you can learn by seeing how JupyterLab users handle them.  Tested by launching a temporary session via MyBinder from here and works in Chrome on a Mac.
Question: I am using Python 3.10.4, GitPython version 3.1.31, mypy version 1.4.1: $ pip show GitPython Name: GitPython Version: 3.1.31 Location: /home/hakon/.pyenv/versions/3.10.4/lib/python3.10/site-packages Requires: gitdb $ python --version Python 3.10.4 $ mypy --version mypy 1.4.1 (compiled: yes)  If run mypy on this minimal example (git-python-types.py) : import git repo = git.Repo('some_dir')  I get the following error: $ mypy --strict git-python-types.py git-python-types.py:3: error: Module "git" does not explicitly export attribute "Repo" [attr-defined] Found 1 error in 1 file (checked 1 source file)  Any ideas on why this error occurs and how to fix it? Some clues I can see the following line in the GitPython source code : from git.repo import Repo # @NoMove @IgnorePep8  but I am not sure if mypy is reading this line or not.

Answer: As documentation suggests this is proper usage https://gitpython.readthedocs.io/en/stable/tutorial.html#meet-the-repo-type from git import Repo ...  then I would consider this a minor bug that should be fixed in GitPython. You can work around this bug by importing it from the submodule git.repo.Repo('some_dir')  or don't use strict. The check it adds here is --no-implicit-reexport, which GitPython currently breaks.
Question: I have installed peewee and postgres. I am using FastAPI as the backend. When I run the request to create tables, it throws the shown error: INFO: 127.0.0.1:33284 - "GET /setup_db HTTP/1.1" 500 Internal Server Error ERROR: Exception in ASGI application Traceback (most recent call last): peewee.ImproperlyConfigured: Postgres driver not installed!  This is my main.py: from fastapi import FastAPI from models.schema import db, User, Plan from datetime import datetime from peewee import * db = PostgresqlDatabase('database_name', host='localhost', port=5432, user='user', password='password') app = FastAPI() @app.get('/') async def Home(): return "Welcome Home" @app.get('/setup_db') async def SetupDB(): db.connect() db.create_tables([User],[Plan]) new_user = User.create( username = 'user1', email = 'user1@email.com', hashed_password = 'pwd####', create_on = datetime(2001, 3, 29, 5, 15, 30) ) print(new_user) return new_user  I tried importing the Models from different folders but it did not work. Please let me know what drivers/config do I need to add to make this work. Thanks & Cheers!

Answer: install postgresql driver: pip install psycopg2  But, I suggest you to use more robust solution, like: sqlalchemy, asyncpg, alembic :)
Question: I am trying to approximate a Bayesian model by keeping the dropout probability during both training and inference (Monte Carlo dropout), in order to obtain the epistemic uncertainty of the model. Is there a way to fix all source of randomness for reproducibility (random seed), but to maintain the randomness of the dropout? # Set random seed for reproducibility seed = 123 torch.manual_seed(seed) random.seed(seed) np.random.seed(seed) # Training and Inference phase (with dropout) dropout_mask = torch.bernoulli(torch.full_like(input, 1 - self.dropout)) skip = self.skip0(input * dropout_mask / (1 - self.dropout)) for i in range(self.layers): residual = x filter = self.filter_convs[i](x) filter = torch.tanh(filter) gate = self.gate_convs[i](x) gate = torch.sigmoid(gate) x = filter * gate dropout_mask = torch.bernoulli(torch.full_like(x, 1 - self.dropout)) x = x * dropout_mask / (1 - self.dropout) s = x s = self.skip_convs[i](s) skip = s + skip if self.gcn_true: x = self.gconv1[i](x, adp) + self.gconv2[i](x, adp.transpose(1, 0)) else: x = self.residual_convs[i](x) x = x + residual[:, :, :, -x.size(3):] if idx is None: x = self.norm[i](x, self.idx) else: x = self.norm[i](x, idx) skip = self.skipE(x) + skip x = F.relu(skip) x = F.relu(self.end_conv_1(x)) x = self.end_conv_2(x) return x  The above code produces same result everytime, which is not what I am trying to do.

Answer: I had to remove this line for things to work as expected. model.eval()  This line seems to disable the dropout during inference.
Question: I am trying to extract data from RDS MySQL instance and load into kinesis data stream using put_record boto3 API. The connection using pymysql is working and I am able to print the table but, I cannot write the data into kinesis data stream. I get this error "Object of type datetime is not JSON serializable". def lambda_handler(event, context): connection = pymysql.connect( host = endpoint, user = username, password = passwrd, database = database_name) cursor = connection.cursor() cursor.execute('SELECT * FROM table LIMIT 10') rows = cursor.fetchall() for row in rows: print("{0} {1} {2}".format(row[0], row[1], row[2])) kinesis = boto3.client('kinesis') response = kinesis.put_record( StreamName="test", Data=json.dumps(rows), PartitionKey="1" ) connection.commit() lambda_handler(None,None)  I tried printing the table and it worked. Only issue is putting records into kinesis data stream.

Answer: This error is thrown when an object with a property of type datetime is passed to json.dumps method. There are several ways on how to resolve it but the quickest one is to convert all datetime properties to string by using the keyword argument of json.dumps import json from datetime import datetime # The default keyword argument can # be set to a function that gets called # for objects that can't otherwise be serialized. response = kinesis.put_record( StreamName="test", Data=json.dumps(rows, default=str), PartitionKey="1" )
Question: Python zip function is its own inverse (in a way), thus we can do this: points = [(1,2), (3,4), (5,6), (7,8)] xs, ys = zip(*points)  and now xs=[1,3,5,7] and ys=[2,4,6,8]. I wonder if something similar can be done with data class instances instead of tuples: from dataclasses import dataclass @dataclass class XY: "2d point" x: float | int y: float | int points = [XY(1,2), XY(3,4), XY(5,6), XY(7,8)] xs, ys = zip(*[(p.x,p.y) for p in points])  but without an explicit list comprehension. Of course, the result would not be a tuple (xs,ys) but a dict with keys x and y because, without an explicit list comprehension, we would be collecting all fields.

Answer: With astuple: from dataclasses import dataclass, astuple @dataclass class XY: "2d point" x: float | int y: float | int def __iter__(self): return iter(astuple(self)) points = [XY(1,2), XY(3,4), XY(5,6), XY(7,8)] xs, ys = zip(*points)  Or instead map it: xs, ys = zip(*map(astuple, points))
Question: I have the following polars dataframe and I wanted to select the last column dynamically. >>> import polars as pl >>> >>> df = pl.DataFrame({ ... "col1": [1, 2], ... "col2": ["2", "3"], ... "col3": [3, 4] ... }) >>> >>> df shape: (2, 3) ┌──────┬──────┬──────┐ │ col1 ┆ col2 ┆ col3 │ │ --- ┆ --- ┆ --- │ │ i64 ┆ str ┆ i64 │ ╞══════╪══════╪══════╡ │ 1 ┆ 2 ┆ 3 │ │ 2 ┆ 3 ┆ 4 │ └──────┴──────┴──────┘ >>> # How to select col3? which is the last column in df  How can i do this in polars?. I can do df.iloc[:,-1:] to select the last column if it's a pandas dataframe. Additional info: >>> import sys >>> sys.version_info sys.version_info(major=3, minor=11, micro=0, releaselevel='final', serial=0) >>> import polars >>> polars.__version__ '0.18.3' 

Answer: To aid in operatons like these, a polars.selectors module was introduced recently. You can simply use last from this module: df.select(cs.last())  shape: (2, 1) ┌──────┐ │ col3 │ │ --- │ │ i64 │ ╞══════╡ │ 3 │ │ 4 │ └──────┘
Question: I don't understand the following behavior of Langchain recursive text splitter. Here is my code and output. from langchain.text_splitter import RecursiveCharacterTextSplitter r_splitter = RecursiveCharacterTextSplitter( chunk_size=10, chunk_overlap=0, # separators=["\n"]#, "\n", " ", ""] ) test = """a\nbcefg\nhij\nk""" print(len(test)) tmp = r_splitter.split_text(test) print(tmp)  Output 13 ['a\nbcefg', 'hij\nk']  As you can see, it outputs chunks of size 7 and 5 and only splits on one of the new line characters. I was expecting output to be ['a','bcefg','hij','k']

Answer: Accord to the split_text funcion in RecursiveCharacterTextSplitter def split_text(self, text: str) -> List[str]: """Split incoming text and return chunks.""" final_chunks = [] # Get appropriate separator to use separator = self._separators[-1] for _s in self._separators: if _s == "": separator = _s break if _s in text: separator = _s break # Now that we have the separator, split the text if separator: splits = text.split(separator) else: splits = list(text) # Now go merging things, recursively splitting longer texts. _good_splits = [] for s in splits: if self._length_function(s) < self._chunk_size: _good_splits.append(s) else: if _good_splits: merged_text = self._merge_splits(_good_splits, separator) final_chunks.extend(merged_text) _good_splits = [] other_info = self.split_text(s) final_chunks.extend(other_info) if _good_splits: merged_text = self._merge_splits(_good_splits, separator) # Here will merge the items if the cusum is less than chunk size in your example is 10 final_chunks.extend(merged_text) return final_chunks  this will merge the items if the cusum is less than chunk size in your example is 10
Question: I have built unittests for my code and everything works fine when running them from vscode. Even running coverage run runs successfully. But when I try to run coverage report I get the following output: No source for code: 'C:\Users\XXX\AppData\Local\Temp\1\__autograph_generated_file4ilniiln.py'  I found out that this happens exactly when I add a test case which contains tensorflow.keras.Model.fit function. If I remove tensorflow.keras.Model.fit then this message does not appear for coverage report command. How can I solve this issue?

Answer: Tensorflow rewrites your code and runs it from a new location. I need some assistance to get it working properly in coverage.py. See this issue for details: https://github.com/nedbat/coveragepy/issues/856 You can try using the -i flag to coverage report to ignore files it can't find. I don't know if you will still get complete data or not.
Question: In Jupyter Notebook we could create slides, like explained here. Here you can see a reproducible example to create Jupyter notebook slides via Anaconda-navigator:  When running this command in the terminal: jupyter nbconvert slides_test.ipynb --to slides --post serve  It will outputs this in your browser:  And for the code cell output:  This is very nice and I would like to use this but in VScode. In the Jupyter notebook of Anaconda navigator we can create slides when clicking on View -> Cell Toolbar -> Slideshow in the header bar. Unfortunately, this is not possible in VScode. So I was wondering if anyone knows how to create Jupyter Notebook slides in VScode?

Answer: Firstly, ensure that you have install Jupyter extension.  It will install several extensions required for Jupyter including slides for you. Then, you can add a slide type to the cell you're on by opening the Command Palette (Cmd+Shift+P) and selecting Switch Slide Type according to the document in GitHub.   You could modify slide types for notebook cells by selecting the slide type on the cell.  After assigning slide types to your cells, create an HTML slideshow presentation by opening the integrated terminal and running the command, jupyter nbconvert '<notebook-file-name>.ipynb' --to slides --post serve.
Question: I have two dataframes df1= ID Value BeginDate EndDate 1 0.5 1/1/12 1/1/13 1 0.6 1/1/13 1/1/14 2 0.4 1/1/12 1/1/13 3 0.7 1/1/12 1/1/13 df2= ID Date 1 6/6/12 1 7/5/12 1 10/5/13 2 8/9/12 3 6/6/12  I want to map the Value column from df1 onto df2 based on the ID column and whether the Date column in df2 fits between BeginDate and EndDate in df1. The problem is that df1 has 10,000 rows and df2 has 1,000,000 rows. I know that this is possible. I am mostly concerned with how to do this quickly. I've had to do something similar before where df1 only has 10 rows, and I used df1.iterrows like so df2['Value'] = 1 for index, row in df1.iterrows(): df1['Value'] = np.where( (row['BeginDate'] <= df2['Date']) & (df2['Date'] <= row['EndDate']), row['Value'], df2['Value'])  However, obviously with df1 now having 10,000 rows, iterrows is no longer preferable. I've considered defining a function and using df2.apply, but this would involve filtering df1 on every iteration. I've had efficiency problems with this approach in the past. It's possible to break up df1 into smaller dataframes based on another column (not displayed in example). So instead of one big dataframe, I'll have 10 smaller ones. I'm thinking of putting them into a dictionary. The keys of this dictionary can be mapped from df2, so this would help, but still leaves me with my initial question. I've seen approaches using a mask and df.loc, but haven't figured out a way with the date conditions. Doing this using pandas isn't essential.

Answer: If the intervals are non overlapping, the most efficient pandas approach would be to merge_asof on the BeginDate, then to check that the EndDate is valid: df1[['BeginDate', 'EndDate']] = df1[['BeginDate', 'EndDate']].apply(pd.to_datetime, format='%d/%m/%y') df2['Date'] = pd.to_datetime(df2['Date'], format='%d/%m/%y') out = (pd.merge_asof( df2.reset_index() .sort_values(by='Date'), df1.sort_values(by='BeginDate'), left_on='Date', right_on='BeginDate', by='ID') .assign(Value=lambda d: d['Value'].where(d['Date'].le(d['EndDate']))) .set_index('index').sort_index() )  Output (with all columns, you can then drop the BeginDate/EndDate):  ID Date Value BeginDate EndDate index 0 1 2012-06-06 0.5 2012-01-01 2013-01-01 1 1 2012-05-07 0.5 2012-01-01 2013-01-01 2 1 2013-05-10 0.6 2013-01-01 2014-01-01 3 2 2012-09-08 0.4 2012-01-01 2013-01-01 4 3 2012-06-06 0.7 2012-01-01 2013-01-01  Alternatively, a more generic method that also handles overlapping intervals would be to use janitor; # pip install janitor import janitor df1[['BeginDate', 'EndDate']] = df1[['BeginDate', 'EndDate']].apply(pd.to_datetime, format='%d/%m/%y') df2['Date'] = pd.to_datetime(df2['Date'], format='%d/%m/%y') df2.conditional_join(df1, ('ID', 'ID', '=='), ('Date', 'BeginDate', '>='), ('Date', 'EndDate', '<='), )
Question: As far as I understood it is preferable to update the column values using .loc method rather than slicing [], because of the potential SettingWithCopy issue. When I use slicing I frequently get these warnings, and I understand that there is a concern when using multindexing. However, when I am using .loc method, my column is not updated when the column's values are used in the right-hand side: a=pd.DataFrame({'A':['2023-01-01','2023-01-02']},[0,1]) a Out[128]: A 0 2023-01-01 1 2023-01-02 a.dtypes Out[130]: A object dtype: object  When I am running either of the following commands to convert object into the datetime, the dtype of the column "A" remains object. Thus there is no effect: a.loc[:,'A']=pd.to_datetime(a.loc[:,'A']) a.loc[:,'A']=pd.to_datetime(a['A'])  When I am using a['A']=pd.to_datetime(a['A'])  My dtype is converted into datetime. I am curious why .loc method does not work in this case? Thanks

Answer: If you use a.loc[:,'A'] the dtype of the Series will not change because you modify only values. You slice your Series (even if you select all rows with :) so the container will remain the same. However, the type of each individual value will change from string to Timestamp. This is the same for any operations where the left part is sliced (not only for DatetimeIndex): b = pd.DataFrame({'B': ['3', '5']}) b.loc[:, 'B'] = pd.to_numeric(b.loc[:, 'B']) # OR b.loc[:, 'B'] = b.loc[:, 'B'].astype(int) # OR b.loc[:, 'B'] = b['B'].astype(int)  Output: >>> b B 0 3 1 5 >>> b.dtypes B object dtype: object >>> b.loc[0, 'B'] 3 >>> type(b.loc[0, 'B']) int  This becomes more obvious if you use a slice like [0:2] or a specific index like 0 with loc. In this case, you understand why the dtype of the container won't change if you change some values and not all values.