eshangj/stackoverflow_q_and_a_sample huggingface dataset:
Question: This code creates a race condition: import threading ITERS = 100000 x = [0] def worker(): for _ in range(ITERS): x[0] += 1 # this line creates a race condition # because it takes a value, increments and then writes # some inrcements can be done together, and lost def main(): x[0] = 0 # you may use `global x` instead of this list trick too t1 = threading.Thread(target=worker) t2 = threading.Thread(target=worker) t1.start() t2.start() t1.join() t2.join() for i in range(5): main() print(f'iteration {i}. expected x = {ITERS*2}, got {x[0]}')  Output: $ python3 test.py iteration 0. expected x = 200000, got 200000 iteration 1. expected x = 200000, got 148115 iteration 2. expected x = 200000, got 155071 iteration 3. expected x = 200000, got 200000 iteration 4. expected x = 200000, got 200000  Python3 version: Python 3.9.7 (default, Sep 10 2021, 14:59:43) [GCC 11.2.0] on linux  I thought GIL would prevent it and not allow two threads run together until they do something io-related or call a C library. At least this is what you may conclude from the docs. Turns out I was wrong. Then, what does GIL actually do, and when do threads run in parallel?

Answer: Reading the docs better, I think there's the answer:  The mechanism used by the CPython interpreter to assure that only one thread executes Python bytecode at a time. This simplifies the CPython implementation by making the object model (including critical built-in types such as dict) implicitly safe against concurrent access. Locking the entire interpreter makes it easier for the interpreter to be multi-threaded, at the expense of much of the parallelism afforded by multi-processor machines.   However, some extension modules, either standard or third-party, are designed so as to release the GIL when doing computationally-intensive tasks such as compression or hashing. Also, the GIL is always released when doing I/O.  I guess this means that each line of source code consists of multiple blocks of bytecode. Bytecode lines/blocks are atomic, i.e. they get executed alone, but the source lines aren't. Here's the byte code that +=1 exapands to (run dis.dis('x[0] += 1') to see):  0 LOAD_NAME 0 (x) 2 LOAD_CONST 0 (0) 4 DUP_TOP_TWO 6 BINARY_SUBSCR 8 LOAD_CONST 1 (1) 10 INPLACE_ADD 12 ROT_THREE 14 STORE_SUBSCR 16 LOAD_CONST 2 (None) 18 RETURN_VALUE  When these lines are executed in concurrent way, race condition occurs. So, GIL does not save you from it. It only prevents race conditions that could damage complex structures like list or dict.
Question: I assume it's possible to schedule a python script to run every day for example, from my github repository. After searching, I've come up with the following main.yml file that resides in the master branch of the repo: the .py file I want to run resides in another branch; I suppose it doesn't have to if it causes an issue, but the script isn't running either way. I'm new to a lot of this, and have a funny feeling I'm missing fundamental pieces to get this working. name: py on: schedule: - cron: "30 11 * * *" #runs at 11:30 UTC everyday jobs: build: runs-on: ubuntu-latest steps: - name: checkout repo content uses: actions/checkout@v2 # checkout the repository content to github runner. - name: setup python uses: actions/setup-python@v2 with: python-version: 3.8 #install the python needed - name: Install dependencies run: | python -m pip install --upgrade pip pip install flake8 pytest if [ -f requirements.txt ]; then pip install -r requirements.txt; fi ref: # branch my_other_branch - name: execute py script # run file run: | python my_file.py 

Answer: Everything seems to be working now, the solution was to move my main.yml file into .github/workflows. I also moved my_file.py from alternate branch into the master branch. One helpful comment recommended specifying the ref branch where you run my_file.py if my_file.py is not located in the default branch.
Question: I have overriden the underscore variable _ in the Python interactive interpreter. How can I make the underscore work again without restarting the interpreter?

Answer: del _  A global _ shadows the builtin _, so deleting the global reveals the builtin again.  It's also worth noting that it doesn't actually stop working, it's just not accessible. You can import builtins to access it: >>> _ = 'foobar' >>> 22 22 >>> _ 'foobar' >>> import builtins >>> 23 23 >>> builtins._ 23
Question: I'm in trouble with a dataframe created from a groupby function. df = base.groupby(['year', 'categ']).agg({'id_prod':'count', 'price':'sum'}).unstack(level=1)  it returns this result :  but I would like to rename id_prod and price to no_sales and revenue but I don't know how to do that because of the MultiIndex with the print(df.columns) the result is : MultiIndex([('id_prod', 0), ('id_prod', 1), ('id_prod', 2), ( 'price', 0), ( 'price', 1), ( 'price', 2)], names=[None, 'categ'])  So is this names=[] I would like to change Thanks for your help !

Answer: df = df.rename(columns={'id_prod': 'no_sales', 'price': 'revenue'}, level=0)  The level=0 indicates where in the multi-index the keys to be renamed can be found.
Question: I would like to plot Feature Importance with SVR, but I don't know if possible with support vector regression it's my code. from sklearn.svm import SVR C=1e3 svr_lin = SVR(kernel="linear", C=C) y_lin = svr_lin.fit(X,Y).predict(X) scores = cross_val_score(svr_lin, X, Y, cv = 5) print(scores) print(scores.mean()) print(scores.std()) 

Answer: SVR does not support native feature importance scores, you might need to try Permutation feature importance which is a technique for calculating relative importance scores that is independent of the model used. First, a model is fit on the dataset, such as a model that does not support native feature importance scores. Then the model is used to make predictions on a dataset, although the values of a feature (column) in the dataset are scrambled. This is repeated for each feature in the dataset. Then this whole process is repeated 3, 5, 10 or more times. The result is a mean importance score for each input feature (and distribution of scores given the repeats). This approach can be used for regression or classification and requires that a performance metric be chosen as the basis of the importance score, such as the mean squared error for regression and accuracy for classification. Permutation feature selection can be used via the permutation_importance() function that takes a fit model, a dataset (train or test dataset is fine), and a scoring function. model = SVR() # fit the model model.fit(X, y) # perform permutation importance results = permutation_importance(model, X, y, scoring='neg_mean_squared_error') # get importance importance = results.importances_mean # summarize feature importance for i,v in enumerate(importance): print('Feature: %0d, Score: %.5f' % (i,v)) # plot feature importance pyplot.bar([x for x in range(len(importance))], importance) pyplot.show()
Question: I have a function in Google Cloud that accepts a number of parameters. I generate ~2k asynchronous requests with different combinations of parameter values using aiohttp: # url = 'https://...' # headers = {'X-Header': 'value'} timeout = aiohttp.ClientTimeout(total=72000000) async def submit_bt(session, url, payload): async with session.post(url, json=payload) as resp: result = await resp.text() async def main(): async with aiohttp.ClientSession(headers=headers, timeout=timeout) as session: tasks = [] gen = payload_generator() # a class that generates dictionaries for payload in gen.param_grid(): tasks.append(asyncio.ensure_future(submit_bt(session, url, payload))) bt_results = await asyncio.gather(*tasks) for result in bt_results: pass asyncio.run(main())  A function takes between 3 to 6 minutes to run, function timeout is set to 9 minutes and maximum number of instances to 3000, but I never see more than 150-200 instances being initiated even when the total number of submitted requests is between 1.5k and 2.5k. On some occasions all requests are processed in 20 to 30 minutes, but sometimes I get an error on the client side: ClientOSError: [Errno 104] Connection reset by peer  that does not correspond to any errors on the server side. I think I should be able to catch it as an aiohttp.client_exceptions.ClientOSError exception, but I am not sure how to handle it in the asynchronous settings, so that the failed request is resubmitted and the premature termination is avoided. Any hints are greatly appreciated.

Answer: The solution suggested by @vaizki in the comment seems to be working well for me. After a closer look at the traceback it turned out that the exception was raised in the submit_bt co-routine, so I added the try-except clause: async def submit_bt(session, url, payload): try: async with session.post(url, json=payload) as resp: result = await resp.text() except aiohttp.client_exceptions.ClientOSError as e: await asyncio.sleep(3 + random.randint(0, 9)) async with session.post(url, json=payload) as resp: result = await resp.text() except Exception as e: result = str(e) return result  It does not look very elegant with the repeated lines, but this is still a work in progress for me and the code structure is not formalized at this stage. Anyway it is clear to see what I wanted to achieve:  post the payload to the function URL, catch exceptions, but repeat the post only in case of ClientOSError and only once.  I did not want to go with a while True kind of loop to avoid infinite execution in case of some serious issues. I tried this code as is a couple of times and I know it worked through a couple of connection resets until the end of the task list, because I got all the results generated by the function, so even in this form it is robust enough for my circumstances.
Question: I have a procedure that returns a recordset using the cursor method: CREATE OR REPLACE PROCEDURE myschema.permissions_sp(rs_out INOUT refcursor) LANGUAGE plpgsql SECURITY DEFINER AS $$ BEGIN OPEN rs_out FOR select schema_name,schema_owner,grantee_type,grantee,p_usage,p_create,object_name,perms,p_select,p_update,p_insert,p_delete,p_truncate,p_references,p_trigger,p_rule from xxxx.myview; END; $$ / GRANT EXECUTE on PROCEDURE myschema.permissions_sp(INOUT refcursor) TO xxxx_user /  And I can call it perfectly fine from workbench using my admin login. BEGIN; CALL meta.permissions_sp('apples'); COMMIT;  Result is a dataset of 16columns x >7k rows (To be honest I don't even need to do the transaction parts, it'll run just fine with the CALL only) However, when I call it via psycopg2, this is what happens : cur = conn.cursor() cur.execute("CALL meta.permissions_sp('apples');") conn.commit() rows = cur.fetchall() print('rows:%s' % (rows))  Output: > rows:[('apples',)]  I've played around with using the commit and not using it. Just really struggling to understand what is going on. At this stage not sure if it's how I'm calling from Python, or on the Redshift side of things. Any guidance appreciated !

Answer: The procedure receives a name as its argument and returns a server-side cursor with that name. On the client side, after calling the procedure you must declare a named cursor with the same name and use it to access the query results. You must do this before committing the connection, otherwise the server-side cursor will be destroyed. with psycopg2.connect(dbname='test') as conn: cur = conn.cursor() cur.execute("""CALL myschema.permissions_sp('mycursor')""") # Use the name that was passed to the procedure. named_cursor = conn.cursor('mycursor') for row in named_cursor.fetchall(): print(row)  This is analogous to how you might get the results in the psql console, as described in the Redshift docs: BEGIN; CALL myschema.permissions_sp('mycursor'); FETCH ALL FROM mycursor; COMMIT;
Question: TLDR I want to kill a subprocess like top while it is still running I am using Fastapi to run a command on input. For example if I enter top my program runs the command but since it does not return, at the moment I have to use a time delay then kill/terminate it. However I want to be able to kill it while it is still running. However at the moment it won't run my kill command until the time runs out. Here is the current code for running a process: @app.put("/command/{command}") async def run_command(command: str): subprocess.run([command], shell=True, timeout=10) return {"run command"}  and to kill it @app.get("/stop") async def stop(): proc.kill() return{"Stop"}  I am new to fastapi so I would be grateful for any help

Answer: It's because subprocess.run is blocking itself - you need to run shell command in background e.g. if you have asnycio loop already on, you could use subprocesses import asyncio process = None @app.get("/command/{command}") async def run_command(command: str): global process process = await asyncio.create_subprocess_exec( command, stdout=asyncio.subprocess.PIPE, stderr=asyncio.subprocess.PIPE ) return {"run command"} @app.get("/stop") async def stop(): process.kill() return {"Stop"}  Or with Popen from subprocess import Popen process = None @app.get("/command/{command}") async def run_command(command: str): global process process = Popen([command]) # something long running return {"run command"}  Adding the timeout option can be tricky as you do not want to wait until it completes (where you could indeed use wait_for function) but rather kill process after specific time. As far I know the best option would be to schedule other process which is responsible for killing main one. The code with asyncio could look like that: @app.get("/command/{command}") async def run_command(command: str): global process process = await asyncio.create_subprocess_exec( command, stdout=asyncio.subprocess.PIPE, stderr=asyncio.subprocess.PIPE ) loop = asyncio.get_event_loop() # We schedule here stop coroutine execution that runs after TIMEOUT seconds loop.call_later(TIMEOUT, asyncio.create_task, stop(process.pid)) return {"run command"} @app.get("/stop") async def stop(pid=None): global process # We need to make sure we won't kill different process if process and (pid is None or process.pid == pid): process.kill() process = None return {"Stop"}
Question: Async For input/output(IO) bound we need to use async code and django is not async by default, but we can achieve this running gunicorn with the gevent worker and monkey patching: gunicorn --worker-class=gevent --worker-connections=1000 --workers=3 main:app  Gunicorn changelog from 2014 https://docs.gunicorn.org/en/stable/2014-news.html?highlight=monkey#gevent-worker:  fix: monkey patching is now done in the worker    Do i still need to monkey patch my app or it's done by default from a worker ?  How did gevent achieve async functionality for my django code ?  Threads If we have a CPU bound we need to use a gthread worker with threads: gunicorn --workers=5 --threads=2 --worker-class=gthread main:app   If we use this configuration for i/o bound, does it work? When one thread is waiting because of i/o, will the other thread be able to work?  I see the point in (3) (if I'm right) because of the wait time in i/o, but if this is a CPU bound, how in our case will the second thread help us or will it only help if the core is not fully loaded by one thread and there is room for another to run?  Are (3) and (4) useless because of GIL ?  For example, 4 people sending request to server with 1 worker and 4 threads. GIL make sure that only 1 thread works. First man start processing with thread 1 and other 3 are waiting ? What are the threads for, then? 

Answer:  Do i still need to monkey patch my app or it's done by default from a worker ? No need to patch anything in your code. No need to modify codes at all.  How did gevent achieve async functionality for my django code ? gunicorn patches everything.  If we use this configuration for i/o bound, does it work? When one thread is waiting because of i/o, will the other thread be able to work? This configuration works for i/o bound. Threads can switch between themselves at anytime (switching controlled by ultimately the operating system), no matter whether the current thread is doing I/O or CPU-bound computation. Multiple threads can work simultaneously on multi-thread CPUs. In contrast, greenlets are more of coroutines rather than threads. If a coroutine gets blocked by I/O, it actively allows another coroutine to take control of the CPU and do non-I/O stuff.  I see the point in (3) (if I'm right) because of the wait time in i/o, but if this is a CPU bound, how in our case will the second thread help us or will it only help if the core is not fully loaded by one thread and there is room for another to run? For a purely CPU-bounded task on a single-thread CPU, extra threads makes little sense.  Are (3) and (4) useless because of GIL? GIL forbids your Python codes running concurrently, but gunicorn mostly uses its libraries not written in Python. You cannot run your Django codes (in Python) with multiple threads, but the I/O tasks (handled by gunicorn, not in Python) may go concurrently. If you do need CPU utilization, use multiple processes (workers=2 * CPU_THREADS + 1) instead of multiple gthreads, or consider non-CPython interpreters like pypy, which is not constrained by GIL, but may not be compatible with your codes.
Question: As a learning project for Python, I am attempting to read all Excel files in a directory and extract the names of all the sheets. I have been trying several available Python modules to do this (pandas in this example), but am running into an issue with most of them depending on openpyxl. This is my current code: import os import pandas directory_root = 'D:\\testFiles' # Dict to hold all files, stats all_files = {} for _current_path, _dirs_in_path, _files_in_path in os.walk(directory_root): # Add all files to this `all_files` for _file in _files_in_path: # Extract filesystem stats from the file _stats = os.stat(os.path.join(_current_path, _file)) # Add the full file path and its stats to the `all_files` dict. all_files[os.path.join(_current_path, _file)] = _stats # Loop through all found files to extract the sheet names for _file in all_files: # Open the workbook xls = pandas.ExcelFile(_file) # Loop through all sheets in the workbook for _sheet in xls.sheet_names(): print(_sheet)  This raises an error from openpyxl when calling pandas.ExcelFile(): ValueError: Max value is 14. From what I can find online, this is because the file contains a font family above 14. How do I read from an Excel (xlsx) file while disregarding any existing formatting? The only potential solution I could find suggests modifying the original file and removing the formatting, but this is not an option as I do not want to modify the files in any way. Is there another way to do this that doesn't have this formatting limitation?

Answer: The issue is that your file does not conform to the Open Office specification. Only certain font families are allowed. Once openpyxl encounters a font out of specification, it throws this error because OpenPyxl only allows spec-conforming excel files. Some Excel readers may not have an issue with this and are more flexible with non-OpenOffice-spec-conforming files, but openpyxl only implements the Apache Open Office spec. The xml being parsed will contain information about the font like this: <font> <b/> <sz val="11"/> <color rgb="FF000000"/> <name val="Century Gothic"/> <family val="34"/> </font>  If the family value is over 14, openpyxl throws this ValueError. There is an underlying descriptor in Open Office that controls this. When other readers like, say, Microsoft Office 365 Excel encounters this, it will change the font family when loading the file to a compliant font (the default, Calibri). As a workaround, if you don't want to change the value (as Microsoft Excel does), you can monkeypatch the descriptor to allow a larger max font family. # IMPORTANT, you must do this before importing openpyxl from unittest import mock # Set max font family value to 100 p = mock.patch('openpyxl.styles.fonts.Font.family.max', new=100) p.start() import openpyxl openpyxl.open('my-bugged-worksheet.xlsx') # this works now!  This can be reproduced using this excel workbook. Before the patch this will fail to load. After the patch, it loads without error.
Question: I am trying to show the z items as text on a Plotly heatmap. I am using the latest version (5.5.0) and following the exact example shown on the Plotly Heatmaps webpage (https://plotly.com/python/heatmaps/), see the section "Text on Heatmap Points" near the bottom. My code is their example code, which is: figHeatmap = go.Figure( data=go.Heatmap( z=[[1,20,30], [20,1,60], [30,60,1]], text=[['1','2','3'],['4','5','6'],['7','8','9']], texttemplate="%{text}", textfont={"size":20} ) )  When I render my app in a webpage, what I see is this:  But on their site it shows with the labels:  I don't get any errors when I run my app, and the labels do indeed show when I hover over the heatmap, but the text is not displaying on the heatmap as it should. Since I'm using their exact example, I don't understand what could be going wrong. Any thoughts? [EDIT]: It seems like this might be an issue with Dash. When I run the example interactively, I do indeed get the text labels. But my application is part of a Dash app, and that is where I am not seeing the rendered labels.

Answer:  This is a new feature in 5.5.0 https://github.com/plotly/plotly.py/releases. After installing and restarting my jupyter kernel, this did not work. Required restart of complete jupyter environment the documented example using go https://plotly.com/python/heatmaps/#text-on-heatmap-points plus note in release notes auto_text parameter, I explored this. below shows use of px and also go by adding numbers as text as well   import plotly.express as px import plotly.graph_objects as go import inflect p = inflect.engine() df = px.data.medals_wide(indexed=True) fig = px.imshow(df, text_auto=True) fig2 = go.Figure(fig.data, fig.layout) fig2 = fig2.update_traces(text=df.applymap(p.number_to_words).values, texttemplate="%{text}", hovertemplate=None) fig.show() fig2.show()   dash  see https://dash.plotly.com/external-resources Controlling the Plotly.js Version Used by dcc.Graph dash 2.0.0 is packaged with a version of plotly.js that does not yet support texttemplate code below shows how to override  import numpy as np import dash import plotly.graph_objects as go import plotly.express as px from dash.dependencies import Input, Output, State from jupyter_dash import JupyterDash import inflect # Build App app = JupyterDash(__name__, external_scripts=["https://cdn.plot.ly/plotly-2.8.3.min.js"]) app.scripts.config.serve_locally = False app.layout = dash.html.Div( [ dash.dcc.Interval(id="run", max_intervals=1), dash.dcc.Graph(id="fig"), ] ) def hm(): p = inflect.engine() df = px.data.medals_wide(indexed=True) fig = px.imshow(df, text_auto=True) fig2 = go.Figure(fig.data, fig.layout) fig2 = fig2.update_traces( text=df.applymap(p.number_to_words).values, texttemplate="%{text}", hovertemplate=None, ) return fig2.update_layout(title=f"dash: {dash.__version__}") @app.callback(Output("fig", "figure"), Input("run", "n_intervals")) def createFig(n): return hm() # Run app and display result inline in the notebook app.run_server(mode="inline")