eshangj/stackoverflow_q_and_a_sample huggingface dataset:
Question: I have the following in test.py: def f(x: int) -> float: pass if __name__=="__main__": f(4)  When I run mypy --strict test.py, I get no errors. I expected mypy would be able to infer that there is a problem with my definition f. It obviously has no return statement and can never return a float. I feel like there is something fundamental which I do not understand here. The presence (or not) of a return statement is something that can be checked statically. Why does mypy miss it?

Answer: The syntax you use is recognized as a function stub, not as a function implementation. Normally, a function stub is written as: def f(x: int) -> float: ...  but this is merely a convenience for def f(x: int) -> float: pass  From the mypy documentation:  Function bodies cannot be completely removed. By convention, we replace them with ... instead of the pass statement.  As mypy does no checks for the function body for stubs, you get no error in this case. And as pointed out by @domarm in the comments, adding any statement besides pass (even a second pass statement) to the function body will result in the expected mypy error.
Question: I am trying to achieve a calculation involving geometric progression (split). Is there any effective/efficient way of doing it. The data set has millions of rows. I need the column "Traded_quantity"       Marker Action Traded_quantity     2019-11-05 09:25 0  0    09:35 2 BUY 3    09:45 0  0    09:55 1 BUY 4    10:05 0  0    10:15 3 BUY 56    10:24 6 BUY 8128     turtle = 2 (User defined) base_quantity = 1 (User defined)  def turtle_split(row): if row['Action'] == 'BUY': return base_quantity * (turtle ** row['Marker'] - 1) // (turtle - 1) else: return 0 df['Traded_quantity'] = df.apply(turtle_split, axis=1).round(0).astype(int)  Calculation For 0th Row, Traded_quantity should be zero (because the Marker is zero) For 1st Row, Traded_quantity should be (1x1) + (1x2) = 3 (Marker 2 will be split into 1 and 1, First 1 will be multiplied with the base_quantity>>1x1, Second 1 will be multiplied with the result from first 1 times turtle>>1x2), then we make a sum of these two numbers) For 2nd Row, Traded_quantity should be zero (because the Marker is zero) For 3rd Row, Traded_quantity should be (2x2) = 4(Marker 1 will be multiplied with the last split from row 1 time turtle i.e 2x2) For 4th Row, Traded_quantity should be zero(because the Marker is zero) For 5th Row, Traded_quantity should be (4x2)+(4x2x2)+(4x2x2x2) = 56(Marker 3 will be split into 1,1 and 1, First 1 will be multiplied with the last split from row3 times turtle >>4x2, Second 1 will be multiplied with the result from first 1 with turtle>>8x2), third 1 will be multiplied with the result from second 1 with turtle>>16x2) then we make a sum of these three numbers) For 6th Row, Traded_quantity should be (32x2)+(32x2x2)+(32x2x2x2)+(32x2x2x2x2)+(32x2x2x2x2x2) = 8128 Whenever there will be a BUY, the traded quantity will be calculated using the last batch from Traded_quantity times turtle. Turns out the code is generating correct Traded_quantity when there is no zero in Marker. Once there is a gap with a couple of zeros geometric progression will not help, I would require the previous fig(from Cache) to recalculate Traded_q. tried with lru_cache for recursion, didn't work.

Answer: This should work def turtle_split(row): global base_quantity if row['Action'] == 'BUY': summation = base_quantity * (turtle ** row['Marker'] - 1) // (turtle - 1) base_quantity = base_quantity * (turtle ** (row['Marker'] - 1))*turtle return summation else: return 0
Question: I have a function that is running inside a for loop. My goal is to run it every day at a specific time, such as 10 am. My code is: def get_sql_backup_transfer(ip, folder,path_sec_folder,sec_folder,path): call(["robocopy",f'{ip}\\{folder}', f'{path_sec_folder}{sec_folder}',"/tee","/r:5","/w:120","/S","/MIR",f"/LOG:{path}{sec_folder}.log"]) for i in sqlserverList : get_sql_backup_transfer(i['ip'] , i['folder'] , path_sec_folder ,i['sec_folder'] , path )  How can I run this code automatically every day at 10 am?

Answer: There are some ways to do this , but the best way is using 'schedule' package, i guess  However, in the first step install the package :  pip install schedule  And then use it in your code like the following codes :  import schedule schedule.every().day.at("10:00").do(yourFunctionToDo,'It is 10:00')
Question: writing a function that should meet a condition on a row basis and return the expected results def bt_quantity(df): df = bt_level(df) df['Marker_change'] = df['Marker'] - df['Marker'].shift(1).fillna(0).round(0).astype(int) df['Action'] = np.where(df['Marker_change'] > 0, "BUY", "") def turtle_split(row): if df['Action'] == 'Buy': return baseQ * (turtle ** row['Marker'] - 1) // (turtle - 1) else: return 0 df['Traded_q'] = df.apply(turtle_split, axis=1).round(0).astype(int) df['Net_q'] = df['Traded_q'].cumsum().round(0).astype(int) print(df.head(39)) return df  This is a common issue, and I am not using any "and" or "or" in the code. still getting the below error ValueError: The truth value of a Series is ambiguous. Use a.empty, a.bool(), a.item(), a.any() or a.all(). I tried changing str to int(BUY >> 1), no progress. P.S. the data set is huge and I am using multiple modules and functions to work on this project.

Answer: As you said, this is really a common problem, you will find the answers from Truth value of a Series is ambiguous. Use a.empty, a.bool(), a.item(), a.any() or a.all() But to address your particular case, this is not a very efficient way of doing it as the dataset is huge. You should use Numpy. That will shorten the runtime drastically. I see two issues with your snippet  There is a Typo. You used "BUY" and then used "Buy". Python is case-sensitive. The col Action is getting tested for "BUY" entirely. The fix is to use row (Not a pythonic way, but a small fix)  def bt_quantity(df): df = bt_level(df) df['Marker_change'] = df['Marker'] - df['Marker'].shift(1).fillna(0).round(0).astype(int) df['Action'] = np.where(df['Marker_change'] > 0, "BUY", "") def turtle_split(row): if row['Action'] == 'BUY': return baseQ * (turtle ** row['Marker'] - 1) // (turtle - 1) else: return 0 df['Traded_q'] = df.apply(turtle_split, axis=1).round(0).astype(int) df['Net_q'] = df['Traded_q'].cumsum().round(0).astype(int) print(df.head(39)) return df
Question: We have Beam data pipeline running on GCP dataflow written using both Python and Java. In the beginning, we had some simple and straightforward python beam jobs that works very well. So most recently we decided to transform more java beam to python beam job. When we having more complicated job, especially the job requiring windowing in the beam, we noticed that there is a significant slowness in python job than java job which end up using more cpu and memory and cost much more. some sample python code looks like:  step1 = ( read_from_pub_sub | "MapKey" >> beam.Map(lambda elem: (elem.data[key], elem)) | "WindowResults" >> beam.WindowInto( beam.window.SlidingWindows(360,90), allowed_lateness=args.allowed_lateness, ) | "GroupById" >> beam.GroupByKey()  And Java code is like:  PCollection<DataStructure> step1 = message .apply( "MapKey", MapElements.into( TypeDescriptors.kvs( TypeDescriptors.strings(), TypeDescriptor.of(DataStructure.class))) .via(event -> KV.of(event.key, event))) .apply( "WindowResults", Window.<KV<String, CustomInterval>>into( SlidingWindows.of(Duration.standardSeconds(360)) .every(Duration.standardSeconds(90))) .withAllowedLateness(Duration.standardSeconds(this.allowedLateness)) .discardingFiredPanes()) .apply("GroupById", GroupByKey.<String, DataStructure>create())  We noticed Python is always using like 3 more times CPU and memory than Java needed. We did some experimental tests that just ran JSON input and JSON output, same results. We are not sure that is just because Python, in general, is slower than java or the way the GCP Dataflow execute Beam Python and Java is different. Any similar experience, tests and reasons why this is are appreciated.

Answer: Yes, this is a very normal performance factor between Python and Java. In fact, for many programs the factor can be 10x or much more. The details of the program can radically change the relative performance. Here are some things to consider:  Profiling the Dataflow job (official docs) Profiling a Dataflow pipeline (medium blog) Profiling Apache Beam Python pipelines (another medium blog) Profiling Python (general Cloud Profiler docs) How can I profile a Python Dataflow job? (previous StackOverflow question on profiling Python job)  If you prefer Python for its concise syntax or library ecosystem, the approach to achieve speed is to use optimized C libraries or Cython for the core processing, for example using pandas/numpy/etc. If you use Beam's new Pandas-compatible dataframe API you will automatically get this benefit.
Question: Let's run the following code: st = {3, 1, 2} st >>> {1, 2, 3} st.pop() >>> 1 st.pop() >>> 2 st.pop() >>> 3  Although sets are said to be unordered, this set behaves as if it was sorted in ascending order. The method pop(), that should return an 'arbitrary element', according to the documentation, returns elements in ascending order as well. What is the reason for this?

Answer: The order correlates to the hash of the object, size of the set, binary representation of the number, insertion order and other implementation parameters. It is completely arbitrary and shouldn't be relied upon: >>> st = {3, 1, 2,4,9,124124,124124124124,123,12,41,15,} >>> st {1, 2, 3, 4, 9, 41, 12, 15, 124124, 123, 124124124124} >>> st.pop() 1 >>> st.pop() 2 >>> st.pop() 3 >>> st.pop() 4 >>> st.pop() 9 >>> st.pop() 41 >>> st.pop() 12 >>> {1, 41, 12} {1, 12, 41} >>> {1, 9, 41, 12} {1, 12, 9, 41} # Looks like 9 wants to go after 12. >>> hash(9) 9 >>> hash(12) 12 >>> hash(41) 41 >>> {1, 2, 3, 4, 9, 41, 12} {1, 2, 3, 4, 9, 12, 41} # 12 before 41 >>> {1, 2, 3, 4, 9, 41, 12, 15} # add 15 at the end {1, 2, 3, 4, 9, 41, 12, 15} # 12 after 41
Question: I was working in my project related to pyqt5 GUI. I used multiprocessing to make it faster. When I run my program in editor, it works fine. But when I converted my program into executable using pyinstaller, then while running the program through this executable, it's not working. GUI opens, but close once it comes to the multiprocessing portion of the code (I get to know this by putting some print statement) I have also tried mutiprocessing.freeze_support(), still it's not worked. if I remove the multiprocessing, program through executable works fine, But I need to use multiprocessing to make it faster. Any suggestion?

Answer: I had the same problem a while ago, and I recommend using Nuitka hence it supports Multiprocessing. If the problem lasts, try to use the threading library: from threading import Thread def worker(a,b): while True: print(a+b) a+=1 my_thread = Thread(target = worker, args = [10,20]) my_thread.start()  You can also setup a class if you want to terminate the thread at a certain point.
Question: I started a Docker container with: docker run -d --shm-size="4g" --hostname selenium_firefox selenium/standalone-firefox In another container with Python: ... >>> driver = webdriver.Remote(command_executor="http://" +selenium_host+":4444/w d/hub", desired_capabilities=DesiredCapabilities.FIREFOX, keep_alive=True) >>> driver.title '' >>> driver.title Traceback (most recent call last): File "<stdin>", line 1, in <module> File "/usr/local/lib/python3.10/site-packages/selenium/webdriver/remote/webdri ver.py", line 447, in title resp = self.execute(Command.GET_TITLE) File "/usr/local/lib/python3.10/site-packages/selenium/webdriver/remote/webdri ver.py", line 424, in execute self.error_handler.check_response(response) File "/usr/local/lib/python3.10/site-packages/selenium/webdriver/remote/errorh andler.py", line 247, in check_response raise exception_class(message, screen, stacktrace) selenium.common.exceptions.WebDriverException: Message: Unable to execute reques t for an existing session: Unable to find session with ID: 5c619451-8361-4ec9-9b 7e-58b7afac15ff Build info: version: '4.1.1', revision: 'e8fcc2cecf' System info: host: 'selenium_firefox', ip: '172.17.0.3', os.name: 'Linux', os.ar ch: 'amd64', os.version: '5.4.0-89-generic', java.version: '11.0.13' Driver info: driver.version: unknown  The first driver.title I ran it immediately after creating the remote webdriver. Then I waited for some time (around 15 minutes) and ran driver.title again, and it seem that the Python console has lost connection to the corresponding browser. Why does this happen and how do I avoid it? It doesn't happen if I don't use a remote webdriver.

Answer: Option 1: Override Docker Selenium Grid default session timeout From docker/selenium documentation:  Grid has a default session timeout of 300 seconds, where the session can be on a stale state until it is killed. You can use SE_NODE_SESSION_TIMEOUT to overwrite that value in seconds.  docker run -d -e SE_NODE_SESSION_TIMEOUT=1000 --shm-size="4g" --hostname selenium_firefox selenium/standalone-firefox  Option 2: Ping your session once in 60 (any < 300) seconds You may execute some driver command in a loop during the idle time for x in range(15): time.sleep(60) driver.current_url   Reference https://github.com/SeleniumHQ/docker-selenium#grid-url-and-session-timeout
Question: When I run this code, I get an error message saying "Error: The dash_bootstrap_components.Row component (version 1.0.2) received an unexpected keyword argument: no_gutters Allowed arguments: align, children, className, class_name, id, justify, key, loading_state, style" I believe it's because of the dash bootstrap version I use. How do I modify my code to make it work? app.layout = dbc.Container([ dbc.Row( dbc.Col(html.H1("My Dashboard", className='text-center'), width=12) ), dbc.Row([ dbc.Col([ dcc.Dropdown(id='my-dpdn', multi=False, value='A', options=[{'label':x, 'value':x} for x in sorted(df['Value'].unique())], ), dcc.Graph(id='line-fig', figure={}) ],# width={'size':5, 'offset':1, 'order':1}, xs=12, sm=12, md=12, lg=5, xl=5 ), dbc.Col([ dcc.Dropdown(id='my-dpdn2', multi=True, value=['B','C'], options=[{'label':x, 'value':x} for x in sorted(df['Value'].unique())], ), dcc.Graph(id='line-fig2', figure={}) ], #width={'size':5, 'offset':0, 'order':2}, xs=12, sm=12, md=12, lg=5, xl=5 ), ], no_gutters=True, justify='start') ], fluid=True) 

Answer: As of Version 1.0 for Dash-bootstrap-components, the no_gutters attribute for the Row component has been deprecated. You are probably using code from a version <= 0.13, read this migration guide for the full details of the changes. You have a couple of options here:  Reading that migration guide will suggest the new way of removing gutters under the heading Row without 'gutters'   Dropped no_gutters prop. Use gutter modifier classes instead. See the docs for examples.  2- You can revert your dash-bootstrap-components library back to 0.13 (eg. pip install dash-bootstrap-components==0.13), but this is not recommended.
Question: When type annotating a variable of type dict, typically you'd annotate it like this: numeralToInteger: dict[str, int] = {...}  However I rewrote this using a colon instead of a comma: numeralToInteger: dict[str : int] = {...}  And this also works, no SyntaxError or NameError is raised. Upon inspecting the __annotations__ global variable: colon: dict[str : int] = {...} comma: dict[str, int] = {...} print(__annotations__)  The output is: {'colon': dict[slice(<class 'str'>, <class 'int'>, None)], 'comma': dict[str, int]}  So the colon gets treated as a slice object and the comma as a normal type hint. Should I use the colon with dict types or should I stick with using a comma? I am using Python version 3.10.1.

Answer: If you have a dictionary whose keys are strings and values are integers, you should do dict[str, int]. It's not optional. IDEs and type-checkers use these type hints to help you. When you say dict[str : int], it is a slice object. Totally different things. Try these in mypy playground: d: dict[str, int] d = {'hi': 20} c: dict[str: int] c = {'hi': 20}  message: main.py:4: error: "dict" expects 2 type arguments, but 1 given main.py:4: error: Invalid type comment or annotation main.py:4: note: did you mean to use ',' instead of ':' ? Found 2 errors in 1 file (checked 1 source file)  Error messages are telling everything
Question: I'm currently doing some work in a server (Ubuntu) without admin rights nor contact with the administrator. When using the help(command) in the python command line I get an error. Here's an example: >>> help(someCommand) /bin/sh: most: command not found  So, this error indicates that most pager is not currently installed. However, the server I'm working on has "more" and "less" pagers installed. So, how can I change the default pager configuration for this python utility?

Answer: This one is annoyingly difficult to research, but I think I found it. The built-in help generates its messages using the standard library pydoc module (the module is also intended to be usable as a standalone script). In that documentation, we find:  When printing output to the console, pydoc attempts to paginate the output for easier reading. If the PAGER environment variable is set, pydoc will use its value as a pagination program.  So, presumably, that's been set to most on your system. Assuming it won't break anything else on your system, just unset or change it. (It still pages without a value set - even on Windows. I assume it has a built-in fallback.)
Question: I have the following: type of X is: <class 'list'> X: [tensor([[1.3373, 0.5666, 0.2337, ..., 0.4899, 0.1876, 0.5892], [0.0320, 0.0797, 0.0052, ..., 0.3405, 0.0000, 0.0390], [0.1305, 0.1281, 0.0021, ..., 0.6454, 0.1964, 0.0493], ..., [0.2635, 0.0237, 0.0000, ..., 0.6635, 0.1376, 0.2988], [0.0241, 0.5464, 0.1263, ..., 0.5766, 0.2352, 0.0140], [0.1740, 0.1664, 0.0057, ..., 0.6056, 0.1020, 1.1573]], device='cuda:0')]  However, following the instructions in this video, I get this error:  X_tensor = torch.FloatTensor(X) ValueError: only one element tensors can be converted to Python scalars  I have the following conversion code: X_tensor = torch.FloatTensor(X)  How can I fix this problem? $ python Python 3.8.10 (default, Nov 26 2021, 20:14:08) [GCC 9.3.0] on linux Type "help", "copyright", "credits" or "license" for more information. >>> import torch >>> torch.__version__ '1.10.1+cu113'  X here is a list of torch.tensors.

Answer: Use torch.stack(): X = torch.stack(X)
Question: I'm writing a function that returns a tuple and using type annotations. To do so, I've written: def foo(bar) -> Tuple[int, int]: pass  This runs, but I've been getting a warning that says:  "Tuple" is not defined Pylance report UndefinedVariable  Given that I'm writing a number of functions that return a tuple type, I'd like to get rid of the warning. I'm assuming I just need to import the package Tuple refers to, but what is the right Python package for it? From my research, I'm inclined to think it's the typing package, but I'm not sure.

Answer: Depending on your exact python version there might be subtle differences here, but what's bound to work is from typing import Tuple def foo(bar) -> Tuple[int, int]: pass  Alternatively I think since 3.9 or 3.10 you can just outright say tuple[int, int] without importing anything.
Question: So my issue is I run this simple code to attempt to make a pytube stream object... from pytube import YouTube yt = YouTube('https://www.youtube.com/watch?v=aQNrG7ag2G4') stream = yt.streams.filter(file_extension='mp4')  And end up with the error in the title. Full error: Traceback (most recent call last): File ".\test.py", line 4, in <module> stream = yt.streams.filter(file_extension='mp4') File "C:\Users\logan\AppData\Local\Programs\Python\Python38\lib\site-packages\pytube\__main__.py", line 292, in streams return StreamQuery(self.fmt_streams) File "C:\Users\logan\AppData\Local\Programs\Python\Python38\lib\site-packages\pytube\__main__.py", line 184, in fmt_streams extract.apply_signature(stream_manifest, self.vid_info, self.js) File "C:\Users\logan\AppData\Local\Programs\Python\Python38\lib\site-packages\pytube\extract.py", line 409, in apply_signature cipher = Cipher(js=js) File "C:\Users\logan\AppData\Local\Programs\Python\Python38\lib\site-packages\pytube\cipher.py", line 33, in __init__ raise RegexMatchError( pytube.exceptions.RegexMatchError: __init__: could not find match for ^\w+\W  Extra data: python version: 3.8.10 pytube version: 11.0.2

Answer: As juanchosaravia suggested on https://github.com/pytube/pytube/issues/1199, in order to solve the problem, you should go in the cipher.py file and replace the line 30, which is: var_regex = re.compile(r"^\w+\W")  With that line: var_regex = re.compile(r"^\$*\w+\W")  After that, it worked again.
Question: I have encountered strange redirect behaviour after returning a RedirectResponse object events.py router = APIRouter() @router.post('/create', response_model=EventBase) async def event_create( request: Request, user_id: str = Depends(get_current_user), service: EventsService = Depends(), form: EventForm = Depends(EventForm.as_form) ): event = await service.post( ... ) redirect_url = request.url_for('get_event', **{'pk': event['id']}) return RedirectResponse(redirect_url) @router.get('/{pk}', response_model=EventSingle) async def get_event( request: Request, pk: int, service: EventsService = Depends() ): ....some logic.... return templates.TemplateResponse( 'event.html', context= { ... } )  routers.py api_router = APIRouter() ... api_router.include_router(events.router, prefix="/event")  this code returns the result 127.0.0.1:37772 - "POST /event/22 HTTP/1.1" 405 Method Not Allowed  OK, I see that for some reason a POST request is called instead of a GET request. I search for an explanation and find that the RedirectResponse object defaults to code 307 and calls POST link I follow the advice and add a status redirect_url = request.url_for('get_event', **{'pk': event['id']}, status_code=status.HTTP_302_FOUND)  And get starlette.routing.NoMatchFound  for the experiment, I'm changing @router.get('/{pk}', response_model=EventSingle) to @router.post('/{pk}', response_model=EventSingle) and the redirect completes successfully, but the post request doesn't suit me here. What am I doing wrong? UPD html form for running event/create logic base.html <form action="{{ url_for('event_create')}}" method="POST"> ... </form>  base_view.py @router.get('/', response_class=HTMLResponse) async def main_page(request: Request, activity_service: ActivityService = Depends()): activity = await activity_service.get() return templates.TemplateResponse('base.html', context={'request': request, 'activities': activity}) 

Answer: When you want to redirect to a GET after a POST, the best practice is to redirect with a 303 status code, so just update your code to:  # ... return RedirectResponse(redirect_url, status_code=303)  As you've noticed, redirecting with 307 keeps the HTTP method and body. Fully working example: from fastapi import FastAPI, APIRouter, Request from fastapi.responses import RedirectResponse, HTMLResponse router = APIRouter() @router.get('/form') def form(): return HTMLResponse(""" <html> <form action="/event/create" method="POST"> <button>Send request</button> </form> </html> """) @router.post('/create') async def event_create( request: Request ): event = {"id": 123} redirect_url = request.url_for('get_event', **{'pk': event['id']}) return RedirectResponse(redirect_url, status_code=303) @router.get('/{pk}') async def get_event( request: Request, pk: int, ): return f'<html>oi pk={pk}</html>' app = FastAPI(title='Test API') app.include_router(router, prefix="/event")  To run, install pip install fastapi uvicorn and run with: uvicorn --reload --host 0.0.0.0 --port 3000 example:app  Then, point your browser to: http://localhost:3000/event/form