eshangj/stackoverflow_q_and_a_sample huggingface dataset:
Question: Edit: I found the problem but not sure why this happens. Whenever I query: http://localhost:4001/hello/ with the "/" in the end - I get a proper 200 status response. I do not understand why. Original Post: Whenever I send a query to my app - I keep getting a 307 redirect. How to get my app to return regular status 200 instead of redirecting it through 307 This is the request output: abm | INFO: 172.18.0.1:46476 - "POST /hello HTTP/1.1" 307 Temporary Redirect abm | returns the apples data. nothing special here. abm | INFO: 172.18.0.1:46480 - "POST /hello/ HTTP/1.1" 200 OK  pytest returns: E assert 307 == 200 E + where 307 = <Response [307]>.status_code test_main.py:24: AssertionError  in my root dir: /__init__.py file: from fastapi import FastAPI from fastapi.middleware.cors import CORSMiddleware # from .configs import cors from .subapp import router_hello from .potato import router_potato from .apple import router_apple abm = FastAPI( title = "ABM" ) # potato.add_middleware(cors) abm.add_middleware( CORSMiddleware, allow_origins=["*"], allow_credentials=True, allow_methods=["*"], allow_headers=["*"], ) abm.include_router(router_hello.router) abm.include_router(router_potato.router) abm.include_router(router_apple.router) @abm.post("/test", status_code = 200) def test(): print('test') return 'test'  /subapp/router_hello.py file: router = APIRouter( prefix='/hello', tags=['hello'], ) @router.post("/", status_code = 200) def hello(req: helloBase, apple: appleHeader = Depends(set_apple_header), db: Session = Depends(get_db)) -> helloResponse: db_apple = apple_create(apple, db, req.name) if db_apple: return set_hello_res(db_apple.potato.api, db_apple.name, 1) else: return "null"  in /Dockerfile: CMD ["uvicorn", "abm:abm", "--reload", "--proxy-headers", "--host", "0.0.0.0", "--port", "4001", "--forwarded-allow-ips", "*", "--log-level", "debug"]  I tried with and without "--forwarded-allow-ips", "*" part.

Answer: It happens because the exact path defined by you for your view is yourdomainname/hello/, so when you hit it without / at the end, it first attempts to get to that path but as it is not available it checks again after appending / and gives a redirect status code 307 and then when it finds the actual path it returns the status code that is defined in the function/view linked with that path, i.e status code 200 in your case. You can also read more about the issue here: https://github.com/tiangolo/fastapi/issues/2060#issuecomment-834868906
Question:     C1 C2 C3 C4     A 12 True 89    9 False 77    5 True 23   B 9 True 45    5 True 45    2 False 78   C 11 True 10    8 False 08    12 False 09     C1 & C2 are the multi index. I'm hoping to get a result which gives me only values in C1 which have values both lower than 10 and greater than or equal to 10 in C2. So in the table above C1 - B should go, with the final result should look like this:     C1 C2 C3 C4     A 12 True 89    9 False 77    5 True 23   C 11 True 10    8 False 08    12 False 09     I tried df.loc[(df.C2 < 10 ) & (df.C2 >= 10)] but this didn't work. I also tried: filter1 = df.index.get_level_values('C2') < 10 filter2 = df.index.get_level_values('C2') >= 10 df.iloc[filter1 & filter2] Which I saw suggested on another post that also didn't work. Any one know how to solve this? Thanks

Answer: Use GroupBy.transform with GroupBy.any for test at least one condition match per groups, so possible last filter by m DataFrame: filter1 = df.index.get_level_values('C2') < 10 filter2 = df.index.get_level_values('C2') >= 10 m = (df.assign(filter1= filter1, filter2=filter2) .groupby(level=0)[['filter1','filter2']] .transform('any')) print (m) filter1 filter2 C1 C2 A 12 True True 9 True True 5 True True B 9 True False 5 True False 2 True False C 11 True True 8 True True 12 True True   df = df[m.filter1 & m.filter2] print (df) C3 C4 C1 C2 A 12 True 89 9 False 77 5 True 23 C 11 True 10 8 False 8 12 False 9  Alternative solution: filter1 = df.index.get_level_values('C2') < 10 filter2 = df.index.get_level_values('C2') >= 10 lvl1 = df.index[filter1].remove_unused_levels().levels[0] lvl2 = df.index[filter2].remove_unused_levels().levels[0] df1 = df.loc[set(lvl1).intersection(lvl2)] print (df1) C3 C4 C1 C2 A 12 True 89 9 False 77 5 True 23 C 11 True 10 8 False 8 12 False 9
Question: I'm trying to ask an isinstance question on a user defined type: ConstData = Union[int, str]: from typing import Union, Optional ConstData = Union[int, str] def foo(x) -> Optional[ConstData]: if isinstance(x, ConstData): # <--- this doesn't work # if isinstance(x, (int, str)): <--- this DOES work ... return x return None  Sadly enough, it doesn't work: $ mypy main.py main.py:4: error: Parameterized generics cannot be used with class or instance checks main.py:4: error: Argument 2 to "isinstance" has incompatible type "object"; expected "Union[type, Tuple[Union[type, Tuple[Any, ...]], ...]]" Found 2 errors in 1 file (checked 1 source file)  Is there any elegant way to solve this?

Answer: About: "Still, with if isinstance(x, get_args(ConstData)): return x mypy can't infer that x has the correct return type. It complains it's Any" I think it is a bug in reveal_type(), Mypy undertands the narrowing of x type. As you can see in the following code, it does not raise an error when test() is called with x: def test(k:Union[int, str]) -> None: pass def foo(x: Any) -> Optional[ConstData]: if isinstance(x, get_args(ConstData)): reveal_type(x) # Mypy: Revealed type is "Any" test(x) # Mypy: no error return x # Mypy: no error return None
Question: I have the below code that has been used to download youtube videos. I automatically detect if it's a playlist or single video. However all the sudden it is giving the above error. What can be the problem? import pafy from log import * import tkinter.filedialog import pytube url = input("Enter url :") directory = tkinter.filedialog.askdirectory() def single_url(url,directory): print("==================================================================================================================") video = pafy.new(url) print(url) print(video.title) #logs(video.title,url) file_object = open(directory+"/links.log", "a") file_object.write(video.title +' '+ url + '\n') file_object.close() print('Rating :',video.rating,', Duration :',video.duration,', Likes :',video.likes, ', Dislikes : ', video.dislikes) #print(video.description) best = video.getbest() print(best.resolution, best.extension) best.download(quiet=False, filepath=directory+'/'+video.title+"." + best.extension) print("saved at :", directory, " directory") print("==================================================================================================================") def playlist_func(url,directory): try: playlist = pytube.Playlist(url) file_object = open(directory+"/links.log", "a") file_object.write('Playlist Url :'+ url + '\n') file_object.close() print('There are {0}'.format(len(playlist.video_urls))) for url in playlist.video_urls: single_url(url,directory) except: single_url(url,directory) playlist_func(url,directory) 

Answer: Your issue doesn't have anything to do with your code. Youtube does no longer have a dislike count, they simply removed it. You just have to wait for the pafy package to be updated accordingly, or patch the package locally and remove that part by yourself. Keep in mind there are at least 5 different pull requests open trying to fix it.
Question: I'm using Python 3.7 to store data in a DynamoDB database and encountering the following error message when I try and write an item to the database: Float types are not supported. Use Decimal types instead.  My code: ddb_table = my_client.Table(table_name) with ddb_table.batch_writer() as batch: for item in items: item_to_put: dict = json.loads(json.dumps(item), parse_float=Decimal) # Send record to database. batch.put_item(Item=item_to_put)  "items" is a list of Python dicts. If I print out the types of the "item_to_put" dict, they are all of type str. Thanks in advance for any assistance.

Answer: Ran into the same issue and it turned out that Python was passing a string parameter as something else. The issue went away when I wrapped all of the items in str().
Question: I bought a book which comes with jupyter notebook. In the first chapter, it asks me to install required libraries. It use {sys.executable} -m. I never see it before. what does {sys.executable} and -m do? also why use --user at the end? typically, I just use ! pip install numpy==1.19.2 Anyone can help me understand it? Thank you!  import sys !{sys.executable} -m pip install numpy==1.19.2 --user !{sys.executable} -m pip install scipy==1.6.2 --user !{sys.executable} -m pip install tensorflow==2.4.0 --user !{sys.executable} -m pip install tensorflow-probability==0.11.0 --user !{sys.executable} -m pip install scikit-learn==0.24.1 --user !{sys.executable} -m pip install statsmodels==0.12.2 --user !{sys.executable} -m pip install ta --user 

Answer: Let's split this question up into multiple parts. Part 1 From the Python documentation:  sys.executable A string giving the absolute path of the executable binary for the Python interpreter, on systems where this makes sense. If Python is unable to retrieve the real path to its executable, sys.executable will be an empty string or None.  Formatting that in, we get: ...\python.exe -m pip install <package> --user  Part 2 Also from the docs:  -m <module-name> Search sys.path for the named module and execute its contents as the __main__ module.  This is generally the same as just pip install <package> --user, however if you have multiple versions of Python installed, the wrong version of pip might get invoked. By using -m, a matching version of pip will always be invoked. Part 3 This time from the pip documentation:  Install to the Python user install directory for your platform. Typically ~/.local/, or %APPDATA%\Python on Windows. (See the Python documentation for site.USER_BASE for full details.)  Basically, this means that instead of installing to the normal package directory (which could require administrator privileges), it installs to %APPDATA%\Python, which should always be accessible as it is in your user folder.
Question: Is there a way to type hint a function that takes as argument its return type? I naively tried to do this: # NOTE: # This code does not work! # # If I define `ret_type = TypeVar("ret_type")` it becomes syntactically # correct, but type hinting is still broken. # def mycoerce(data: Any, ret_type: type) -> ret_type return ret_type(data) a = mycoerce("123", int) # desired: a type hinted to int b = mycoerce("123", float) # desired: b type hinted to float  but it doesn't work.

Answer: Have a look at Generics, especially TypeVar. You can do something like this: from typing import TypeVar, Callable R = TypeVar("R") D = TypeVar("D") def mycoerce(data: D, ret_type: Callable[[D], R]) -> R: return ret_type(data) a = mycoerce("123", int) # desired: a type hinted to int b = mycoerce("123", float) # desired: b type hinted to float print(a, b)
Question: I have a pandas dataframe import pandas as pd s = pd.DataFrame({'ioc' : ['ESP', 'CYP', 'USA', 'ESP', 'NED']})  and I want to return out = pd.DataFrame( {'ioc' : ['ESP', 'CYP', 'USA', 'ESP', 'NED'], 'countryName' : ['Spain', 'Cyprus', 'United States of America', 'Spain', 'Netherlands']}) 

Answer: Use List of IOC country codes ioc = pd.read_html('https://en.wikipedia.org/wiki/List_of_IOC_country_codes')[0] ioc = ioc.assign(Code=ioc['Code'].str[-3:]).set_index('Code')['National Olympic Committee'] s['countryName'] = s['ioc'].map(ioc) print(s) # Output: ioc countryName 0 ESP Spain 1 CYP Cyprus 2 USA United States 3 ESP Spain 4 NED Netherlands
Question: I am trying to find the cosine similarity between two columns of type array in a pyspark dataframe and add the cosine similarity as a third column, as shown below     Col1 Col2 Dot Prod     [0.5, 0.6 ... 0.7] [0.5, 0.3 .... 0.1] dotProd(Col1, Col2)     The current implementation I have is: import pyspark.sql.functions as func def cosine_similarity(df, col1, col2): df_cosine = df.select(func.sum(df[col1] * df[col2]).alias('dot'), func.sqrt(func.sum(df[col1]**2)).alias('norm1'), func.sqrt(func.sum(df[col2] **2)).alias('norm2')) d = df_cosine.rdd.collect()[0].asDict() return d['dot']/(d['norm1'] * d['norm2'])  But I guess the above code only for works for columns with integer values. Is there anyway I would be able to extend the above function to achieve a similar behavior for array columns

Answer: For the array of double, you can use the aggregate function. df = spark.createDataFrame([[[0.1, 0.5, 2.0, 1.0], [3.0, 2.4, 0.2, 1.1]]], ['Col1', 'Col2']) df.show() +--------------------+--------------------+ | Col1| Col2| +--------------------+--------------------+ |[0.1, 0.5, 2.0, 1.0]|[3.0, 2.4, 0.2, 1.1]| +--------------------+--------------------+ df.withColumn('dot', f.expr('aggregate(arrays_zip(Col1, Col2), 0D, (acc, x) -> acc + (x.Col1 * x.Col2))')) \ .withColumn('norm1', f.expr('sqrt(aggregate(Col1, 0D, (acc, x) -> acc + (x * x)))')) \ .withColumn('norm2', f.expr('sqrt(aggregate(Col2, 0D, (acc, x) -> acc + (x * x)))')) \ .withColumn('cosine', f.expr('dot / (norm1 * norm2)')) \ .show(truncate=False) +--------------------+--------------------+---+-----------------+-----------------+------------------+ |Col1 |Col2 |dot|norm1 |norm2 |cosine | +--------------------+--------------------+---+-----------------+-----------------+------------------+ |[0.1, 0.5, 2.0, 1.0]|[3.0, 2.4, 0.2, 1.1]|3.0|2.293468988235943|4.001249804748511|0.3269133956691457| +--------------------+--------------------+---+-----------------+-----------------+------------------+
Question: I am working with (lists of) lists of numpy arrays. As a bare bones example, consider this piece of code: a = [np.zeros(5)] b = a.copy() b[0] += 1  Here, I copy a list of one array from a to b. However, the array itself is not copied, so: print(a) print(b)  both give [array([1., 1., 1., 1., 1.])]. If I want to make a copy of the array as well, I could do something like: b = [arr.copy() for arr in a]  and a would remain unchanged. This works well for a simple list, but it becomes more complicated when working with nested lists of arrays where the number of arrays in each list is not always the same. Is there a simple way to copy a multi-level list and every object that it contains without keeping references to the objects in the original list? Basically, I would like to avoid nested loops as well as dealing with the size of each individual sub-list.

Answer: What you are looking for is a deepcopy import numpy as np import copy a = [np.zeros(5)] b = copy.deepcopy(a) b[0] += 1 # a[0] is not changed  This is actually method recommended in numpy doc for the deepcopy of object arrays.
Question: I have the following (typed) Python function: from typing import overload, Literal, NoReturn @overload def check(condition: Literal[False], msg: str) -> NoReturn: pass @overload def check(condition: Literal[True], msg: str) -> None: pass def check(condition, msg): if not condition: raise Exception(msg)  Pyright type-checker complains: Overloaded function implementation is not consistent with signature of overload 1 Function return type "NoReturn" is incompatible with type "None" Type cannot be assigned to type "None"  I'm confused by this-- Pyright apparently can't determine that check will always throw an error if condition is False. How can I massage this to make it work?

Answer: I posted this to the Pyright issue tracker and was informed that I had to add a Union[NoReturn, None] annotation to the check implementation to resolve the error:  [This] falls out of pyright's rules for return type inference. Pyright will infer a NoReturn type only in the case that all code paths raise an exception. It's quite common for some code paths raise an exception, and it would produce many false positives if pyright were to include NoReturn in the union in that case, so NoReturn is always elided from a union in an inferred return type. Mypy doesn't have any support for return type inference, so that explains why this doesn't occur for mypy. The correct way to annotate this is to include an explicit None | NoReturn return type for the implementation.  Unfortunately, this defeats the purpose of the overloads in the first place, which is to allow PyRight to infer from the arguments when NoReturn is the return type. I asked whether it is possible to use overloads to conditionally express NoReturn to the type checker. Apparently it is not:  Unfortunately, pyright cannot make this determination because of its core architecture. The "reachability" of nodes within a code flow graph cannot rely on type evaluations because type evaluations depend on reachability of code flow nodes. To work around this chicken-and-egg problem, the logic that determines reachability does some basic checks to determine if the called function is potentially a "NoReturn" function, but these basic checks are not sophisticated enough to handle overload evaluation. Evaluating overloads requires the full type evaluator, and that depends on reachability.
Question: I have a binary time series classification problem. Since it is a time series, I can't just train_test_split my data. So, I used the object tscv = TimeSeriesSplit() from this link, and got something like this:  I can see from GridSearchCV and cross_val_score that i can pass as parameter my split strategy cv = tscv. But my question is, whats the difference between GridSearchCV and coss_val_score? Using one of them is enough to train/test my model? or should i use both? First the GridSearchCV to get the best hyperparamaters and then the cross_val_score?

Answer: Grid search is a method to evaluate models by using different hyperparameter settings (the values of which you define in advance). Your GridSearch can use cross validation (hence, GridSearchCV exists) in order to deliver a final score for the the different parameter settings of your model. After the training and the evaluation (after the grid search has finished), you can take a look at the parameters with which your model performed best (by taking a look at the attribute best_params_dict).So, Grid search is basically a brute forcing strategy in which you run the model with all possible hyperparameter combinations. With coss_val_score you don't perform the grid search (you don't use the strategy mentioned above with all predefined params), but you get the score after the cross-validation. I hope it is now clear.
Question: I'm trying to create a script which downloads a file from a website and for this I want to change the download filepath. When I try to do this with the Firefox options it gives me this error: TypeError: WebDriver.__init__() got an unexpected keyword argument 'firefox_options'  Code: from selenium import webdriver from selenium import webdriver from selenium.webdriver.common.by import By from selenium.webdriver.firefox.options import Options from selenium.webdriver.common.keys import Keys import time options = Options() options.add_argument("download.default_directory=C:\\Music") browser = webdriver.Firefox(firefox_options=options, executable_path=r'C:\\selenium\\geckodriver.exe') browser.get('https://duckduckgo.com/') 

Answer: The browser option parameter firefox_options was deprecated in Selenium 3.8.0    Browser option parameters are now standardized across drivers as options. firefox_options, chrome_options, and ie_options are now deprecated    Instead you have to use options as follows: from selenium.webdriver.firefox.options import Options options = Options() options.add_argument("download.default_directory=C:\\Music") browser = webdriver.Firefox(options=options, executable_path=r'C:\\selenium\\geckodriver.exe')
Question: e.g. if I write import plotly.express as px df = px.data.tips() fig = px.scatter(df, x="total_bill", y="tip", facet_col="sex")  then how can I get the width and height of fig? If I do fig.layout.width  then I get None.

Answer: That is the correct way to get the width. fig.layout.width is None because the width has not been set yet. If you set it explicitly you can retrieve it fig = px.scatter(df, x="total_bill", y="tip", facet_col="sex", width=200) >>> fig.layout.width 200  If not set explicitly the width is initialised when executing the show method based on defaults defined by plotly.js.  show(*args, **kwargs) Show a figure using either the default renderer(s) or the renderer(s) specified by the renderer argument Parameters ... width (int or float) – An integer or float that determines the number of pixels wide the plot is. The default is set in plotly.js. ...  If we look at the plotly.js documentation we see the default width is 700 and the default height is 450. If you set fig.layout.autosize = False you can see that these defaults are correct. Otherwise the width and height are re-initialised on each relayout.  autosize: Determines whether or not a layout width or height that has been left undefined by the user is initialized on each relayout. Note that, regardless of this attribute, an undefined layout width or height is always initialized on the first call to plot.  https://plotly.com/javascript/reference/layout/#layout-autosize
Question: I want validate a payload schema & I am using Pydantic to do that. The class created by inheriting Pydantic's BaseModel is named as PayloadValidator and it has two attributes, addCustomPages which is list of dictionaries & deleteCustomPages which is a list of strings. class NestedCustomPages(BaseModel): """This is the schema for each custom page.""" html: str label: str type: int class PayloadValidator(BaseModelWithContext): """This class defines the payload load schema and also validates it.""" addCustomPages: Optional[List[NestedCustomPages]] deleteCustomPages: Optional[List[str]]  I want to declare either of the attributes of class PayloadValidator as optional. I have tried looking for the solution for this but couldn't find anything.

Answer: There was a question on this a while ago on the pydantic Github page: https://github.com/samuelcolvin/pydantic/issues/506. The conclusion there includes a toy example with a model that requires either a or b to be filled by using a validator: from typing import Optional from pydantic import validator from pydantic.main import BaseModel class MyModel(BaseModel): a: Optional[str] = None b: Optional[str] = None @validator('b', always=True) def check_a_or_b(cls, b, values): if not values.get('a') and not b: raise ValueError('Either a or b is required') return b mm = MyModel()
Question: If I use GET (given an id) I get a JSON like: { "data": { "id": "81", "ks": { "k1": 25, "k2": 5 }, "items": [ { "id": 1, "name": "John", "surname": "Smith" }, { "id": 2, "name": "Jane", "surname": "Doe" } ] }, "server-time": "2021-12-09 14:18:40" }  with the particular case (if id does not exist): { "data": { "id": -1, "ks": "", "items": [] }, "server-time": "2021-12-10 09:35:22" }  I would like to create a Pydantic model for managing this data structure (I mean to formally define these objects). What is the smartest way to manage this data structure by creating classes (possibly nested)?

Answer: If you don't need data validation that pydantic offers, you can use data classes along with the dataclass-wizard for this same task. It's slightly easier as you don't need to define a mapping for lisp-cased keys such as server-time. Simple example below: from __future__ import annotations from dataclasses import dataclass from datetime import datetime from dataclass_wizard import fromdict @dataclass class Something: data: Data # or simply: # server_time: str server_time: datetime @dataclass class Data: id: int ks: dict[str, int] items: list[Person] @dataclass class Person: id: int name: str surname: str # note: data is defined in the OP above input_data = ... print(fromdict(Something, input_data))  Output: Something(data=Data(id=81, ks={'k1': 25, 'k2': 5}, items=[Person(id=1, name='John', surname='Smith'), Person(id=2, name='Jane', surname='Doe')]), server_time=datetime.datetime(2021, 12, 9, 14, 18, 40))