eshangj/stackoverflow_q_and_a_sample huggingface dataset:
Question: Have a huge pandas dataframe (df) like this:  id date a b c 0 0023 201110132120 -30 -45 7 1 0023 201110132130 -30 11 9111 2 0023 201110132140 -24 44 345 3 0023 201110132150 -19 223 11 4 0023 201110132200 -23 -3456 -1250  I need to write this dataframe to a file with special fixed-width for each field. For this i used numpy, f.e.: np.savetxt('out.txt', df.values, fmt='%+4s %+12s %+5s %+5s %+6s')  That work's fine. Only lost header in this case. Is there a workaround? I tested it also with pandas to_string function: df.to_string()  But it is so slow. Why? Are there other options?

Answer: One option is to abuse header option in savetxt: formats = '%+4s %+12s %+5s %+5s %+6s' headers = [format(str(x),y.replace('%+','>')) for x, y in zip(df.columns,formats.split())] np.savetxt('out.txt', df.values, fmt=formats, header=' '.join(headers), comments='')
Question: I have a fresh EC2 Ubuntu 18.04 I have installed anaconda as it it in the official guide https://docs.anaconda.com/anaconda/install/linux/ apt-get install libgl1-mesa-glx libegl1-mesa libxrandr2 libxrandr2 libxss1 libxcursor1 libxcomposite1 libasound2 libxi6 libxtst6 64 bit installation  But then I have YML files that I have used to create conda environments before on other EC2s and when I ma tiring to setup a new environment with it ubuntu@ip........:~$ conda env create --name my_env_name --file=my_env_file.yml it it gives me the following error Collecting package metadata (repodata.json): / Killed Now I am trying with the following guide

Answer: Adding more RAM helps I switched from 0.5 GB to 8GB RAM than the problem disappears
Question: Is setuptools always installed with Python? I would like to invoke setuptools at runtime outside of a setup.py script. In other words, should I include setuptools inside my package's requirements.txt and setup.py's install_requires list?  Background I have noticed when creating a new virtual environment (with Python 3.7.9) that both pip and setuptools are installed by default: python -m venv venv source ./venv/bin/activate pip list Package Version ---------- ------- pip 20.1.1 setuptools 47.1.0  This is documented here: Creating Virtual Environments:  venv is available by default in Python 3.3 and later, and installs pip and setuptools into created virtual environments in Python 3.4 and later.  Even in a vanilla version of Python 3.7.6 (installed via pyenv), the packages installed by default are both pip and setuptools.  Research Should setuptools be included in setup_requires in Python? Informs that setuptools should not be included in setup_requires, but does not talk about it being included in package requirements for runtime use.

Answer: TL;DR Formally, No. Usually, Yes. the setuptools is not part of the python vanilla codebase, hence not a vanilla modules. python.org installers or mac homebrew will install it for you, but if someone compile the python by himself or install it on some linux distribution he may not get it and will need to install it by himself.
Question: def calcPath(trace_map, x, y): n = len(trace_map) count = 0 if x > n - 1 or y > n - 1: pass elif x < n and y < n: if x + trace_map[x][y] == (n - 1) and y == (n - 1): count += 1 elif x == (n - 1) and y + trace_map[x][y] == (n - 1): count += 1 else: calcPath(trace_map, x + trace_map[x][y], y) calcPath(trace_map, x, y + trace_map[x][y]) return count if __name__ == "__main__": trace_map = [ [1, 2, 9, 4, 9], [9, 9, 9, 9, 9], [9, 3, 9, 9, 2], [9, 9, 9, 9, 9], [9, 9, 9, 1, 0], ] print(calcPath(trace_map, 0, 0)) trace_map = [[1, 1, 1], [1, 1, 2], [1, 2, 0]] print(calcPath(trace_map, 0, 0))  I want to count the existing routes of the given maze. (anyway, the problem itself is not that important) Problem is, I tried to count the number of cases that fit the conditions within the recursive functions. These are two conditions that have to be counted. if x + trace_map[x][y] == (n - 1) and y == (n - 1): if x == (n - 1) and y + trace_map[x][y] == (n - 1):  I tried counting the conditions like this count = 0 if condition = True: count +=1  But since I'm using recursive functions, if I declare count = 0 in the function, the count value stays 0. Shortly, I just want to keep the counter unaffected by the recursive function.

Answer: One of the ways to solve this is by adding the count you get from each recursive function's return. When you call the recursive function, take the count that is returned and add it to the count variable in the current scope. For example: def calcPath(trace_map, x, y): n = len(trace_map) count = 0 if x > n - 1 or y > n - 1: pass elif x < n and y < n: if x + trace_map[x][y] == (n - 1) and y == (n - 1): count += 1 elif x == (n - 1) and y + trace_map[x][y] == (n - 1): count += 1 else: count += calcPath(trace_map, x + trace_map[x][y], y) count += calcPath(trace_map, x, y + trace_map[x][y]) return count  An alternative solution would be to create a global variable and reset it to 0 every time the function is called (although I don't recommend this since it requires ceremony everytime the function is called). That might look something like this: count = 0 # Global variable def calcPath(trace_map, x, y): global count n = len(trace_map) if x > n - 1 or y > n - 1: pass elif x < n and y < n: if x + trace_map[x][y] == (n - 1) and y == (n - 1): count += 1 elif x == (n - 1) and y + trace_map[x][y] == (n - 1): count += 1 else: calcPath(trace_map, x + trace_map[x][y], y) calcPath(trace_map, x, y + trace_map[x][y]) if __name__ == "__main__": trace_map = [ [1, 2, 9, 4, 9], [9, 9, 9, 9, 9], [9, 3, 9, 9, 2], [9, 9, 9, 9, 9], [9, 9, 9, 1, 0], ] print(calcPath(trace_map, 0, 0)) # Use count in some way count = 0 # Reset the count trace_map = [[1, 1, 1], [1, 1, 2], [1, 2, 0]] print(calcPath(trace_map, 0, 0))
Question: Background In a confusion matrix, the diagonal represents the cases that the predicted label matches the correct label. So the diagonal is good, while all other cells are bad. To clarify what is good and what is bad in a CM for non-experts, I want to give the diagonal a different color than the rest. I want to achieve this with Python & Seaborn. Basically I'm trying to achieve what this question does in R (ggplot2 Heatmap 2 Different Color Schemes - Confusion Matrix: Matches in Different Color Scheme than Missclassifications) Normal Seaborn Confusion Matrix with heatmap import numpy as np import seaborn as sns cf_matrix = np.array([[50, 2, 38], [7, 43, 32], [9, 4, 76]]) sns.heatmap(cf_matrix, annot=True, cmap='Blues') # cmap='OrRd'  Which results in this image:  Goal I would like to color the non-diagonal cells with e.g. cmap='OrRd'. So I imagine there would be 2 colorbars, 1 blue for the diagonal and 1 for the other cells. Preferably the values of both colorbars match (so both e.g. 0-70 and not 0-70 and 0-40). How would I approach this? The following is not made with code, but with photo editing software: 

Answer: You can use mask= in the call to heatmap() to choose which cells to show. Using two different masks for the diagonal and the off_diagonal cells, you can get the desired output: import numpy as np import seaborn as sns cf_matrix = np.array([[50, 2, 38], [7, 43, 32], [9, 4, 76]]) vmin = np.min(cf_matrix) vmax = np.max(cf_matrix) off_diag_mask = np.eye(*cf_matrix.shape, dtype=bool) fig = plt.figure() sns.heatmap(cf_matrix, annot=True, mask=~off_diag_mask, cmap='Blues', vmin=vmin, vmax=vmax) sns.heatmap(cf_matrix, annot=True, mask=off_diag_mask, cmap='OrRd', vmin=vmin, vmax=vmax, cbar_kws=dict(ticks=[]))   If you want to get fancy, you can create the axes using GridSpec to have a better layout: import numpy as np import seaborn as sns fig = plt.figure() gs0 = matplotlib.gridspec.GridSpec(1,2, width_ratios=[20,2], hspace=0.05) gs00 = matplotlib.gridspec.GridSpecFromSubplotSpec(1,2, subplot_spec=gs0[1], hspace=0) ax = fig.add_subplot(gs0[0]) cax1 = fig.add_subplot(gs00[0]) cax2 = fig.add_subplot(gs00[1]) sns.heatmap(cf_matrix, annot=True, mask=~off_diag_mask, cmap='Blues', vmin=vmin, vmax=vmax, ax=ax, cbar_ax=cax2) sns.heatmap(cf_matrix, annot=True, mask=off_diag_mask, cmap='OrRd', vmin=vmin, vmax=vmax, ax=ax, cbar_ax=cax1, cbar_kws=dict(ticks=[]))
Question: I want to do a multiple queries. Here is my data frame: data = {'Name':['Penny','Ben','Benny','Mark','Ben1','Ben2','Ben3'], 'Eng':[5,1,4,3,1,2,3], 'Math':[1,5,3,2,2,2,3], 'Physics':[2,5,3,1,1,2,3], 'Sports':[4,5,2,3,1,2,3], 'Total':[12,16,12,9,5,8,12], 'Group':['A','A','A','A','A','B','B']} df1=pd.DataFrame(data, columns=['Name','Eng','Math','Physics','Sports','Total','Group']) df1   I have 3 queries:  Group A or B Math > Eng Name starts with 'B'  I tried to do it one by one df1[df1.Name.str.startswith('B')] df1.query('Math > Eng') df1[df1.Group == 'A'] #I cannot run the code with df1[df1.Group == 'A' or 'B']  Then, I tried to merge those queries df1.query("'Math > Eng' & 'df1[df1.Name.str.startswith('B')]' & 'df1[df1.Group == 'A']") TokenError: ('EOF in multi-line statement', (2, 0))  I also tried to pass str.startswith() into df.query() df1.query("df1.Name.str.startswith('B')") UndefinedVariableError: name 'df1' is not defined  I have tried lots of ways but no one works. How can I put those queries together?

Answer: The long way to solve this – and the one with the most transparency, so best for beginners – is to create a boolean column for each filter. Then sum those columns as one final filter: df1['filter_1'] = df1['Group'].isin(['A','B']) df1['filter_2'] = df1['Math'] > df1['Eng'] df1['filter_3'] = df1['Name'].str.startswith('B') # If all are true df1['filter_final'] = df1[['filter_1', 'filter_2', 'filter_3']].all(axis=1)  You can certainly combine these steps into one: mask = ((df1['Group'].isin(['A','B'])) & (df1['Math'] > df1['Eng']) & (df1['Name'].str.startswith('B')) ) df['filter_final'] = mask  Lastly, selecting rows which satisfy your filter is done as follows: df_filtered = df1[df1['filter_final']]  This selects rows from df1 where final_filter == True
Question: from selenium import webdriver from selenium.webdriver.common.by import By from selenium.webdriver.support.ui import WebDriverWait from selenium.webdriver.support import expected_conditions as EC from selenium.webdriver.firefox.options import Options options = Options() options.headless = True driver = webdriver.Firefox(options=options) driver.get("https://hapondo.qa/rent/doha/apartments/studio") element = WebDriverWait(driver, 10).until( EC.presence_of_element_located((By.XPATH, "/html/head/title")) ) print(element.text)  Unable to get page title under headless option? Tried to wait and even tried driver.title

Answer: You need to take care of a couple of things as follows:  To retrieve the Page Title instead of using a xpath you need to use driver.title The hapondo website contains JavaScript enabled elements.   Solution To extract the Page Title you need to induce WebDriverWait for the title_contains() and you can use either of the following Locator Strategy:  Code Block: from selenium import webdriver from selenium.webdriver.support.ui import WebDriverWait from selenium.webdriver.support import expected_conditions as EC options = webdriver.ChromeOptions() options.add_argument('--headless') options.add_argument('--window-size=1920,1080') driver = webdriver.Chrome(options=options, executable_path=r'C:\WebDrivers\chromedriver.exe') driver.get('https://hapondo.qa/rent/doha/apartments/studio') WebDriverWait(driver, 10).until(EC.title_contains("hapondo")) print(driver.title)   Console Output: Studio Apartments for rent in Doha | hapondo     References You can find a couple of relevant detailed discussions in:  How to make selenium wait before getting contents from the actual website which loads after the landing page through IEDriverServer and IE
Question: I have an Websocket endpoint defined in separate file, like: from starlette.endpoints import WebSocketEndpoint from connection_service import ConnectionService class WSEndpoint(WebSocketEndpoint): """Handles Websocket connections""" async def on_connect(self, websocket: WebSocket, connectionService: ConnectionService = Depends(ConnectionService)): """Handles new connection""" self.connectionService = connectionService ...  and in the main.py I register endpoint as: from fastapi import FastAPI from starlette.routing import WebSocketRoute from ws_endpoint import WSEndpoint app = FastAPI(routes=[ WebSocketRoute("/ws", WSEndpoint) ])  But Depends for my endpoint is never resolved. Is there a way to make it work? Plus, what is even the purpose of this mechanism in FastAPI? Cannot we just use local/global variables?

Answer: TL;DR The documents seem to hint that you can only use Depends for request functions. Explanation I found a related issue #2057 in the FastAPI repo and it seems the Depends(...) only works with the requests and not anything else. I confirmed this by, from fastapi import Depends, FastAPI app = FastAPI() async def foo_func(): return "This is from foo" async def test_depends(foo: str = Depends(foo_func)): return foo @app.get("/") async def read_items(): depends_result = await test_depends() return depends_result In this case, the dependency didn't get resolved.  Coming to your case, you can resolve the dependency something like this, from starlette.endpoints import WebSocketEndpoint from connection_service import ConnectionService class WSEndpoint(WebSocketEndpoint): async def on_connect( self, websocket: WebSocket, connectionService=None ): if connectionService is None: connectionService = ConnectionService() # calling the depend function self.connectionService = connectionService
Question: I am trying to re-write this code, which was written a while ago. It has multiple chuncks, hence, I separated it into smaller pieces and re-write step by step. For instance, converting .ix to iloc, etc. This chunk gives me an error: #Loop through all rows, skip the user column, and fill with similarity scores for i in range(0,len(data_sims.index)): for j in range(1,len(data_sims.columns)): user = data_sims.index[i] product = data_sims.columns[j] if data.iloc[i][j] == 1: data_sims.iloc[i][j] = 0 else: product_top_names = data_neighbours.iloc[product][1:10] product_top_sims = data_ibs.iloc[product].order(ascending=False)[1:10] user_purchases = data_germany.iloc[user,product_top_names] data_sims.iloc[i][j] = getScore(user_purchases,product_top_sims)  Get an error TypeError: Cannot index by location index with a non-integer key  I guess there is something here, which requires updating, but cannot find what exactly. I do not think it is about data. It is just about updating the code. Appreciate any tips!

Answer: I'm not exactly sure what that piece of code is supposed to do (though it seems like there may be a more efficient way of doing it). Also, it looks like you are referencing j outside the j loop. However, the specific error you're getting is mostly likely related to: product_top_names = data_neighbours.iloc[product][1:10]  iloc only works with integers, and I guess product is a string. If that's the case, something like product_top_names = data_neighbours[product].iloc[1:10]  should get rid of the error (assuming product is the name of a column).
Question: How does one pip install with all extras? I'm aware that doing something like: pip install -e .[docs,tests,others]  is an option. But, is it possible to do something like: pip install -e .[all]  This question is similar to setup.py/setup.cfg install all extras. However, the answer there requires that the setup.cfg file be edited. Is it possible to do it without modifying setup.py or setup.cfg?

Answer: pip has a --report= option which we can use: --report <file> Generate a JSON file describing what pip did to install the provided requirements. Can be used in combination with --dry-run and --ignore-installed to 'resolve' the requirements. When - is used as file name it writes to stdout. When writing to stdout, please combine with the --quiet option to avoid mixing pip logging output with JSON output.  So let's install the package once without extras and with the --report=- option to get JSON on standard output. Pipe the JSON to jq, digging out the extras defined for the package. Then install the package again with all extras if any exist. (You could also --dry-run the first step to postpone actual installation until the last step). EXTRAS=$(\ pip install --quiet --report=- --editable="." \ | jq --raw-output '.install[0].metadata.provides_extra|join(",")' 2> /dev/null \ ) [ -n "${EXTRAS}" ] && pip install --editable=".[${EXTRAS}]"
Question: I'm trying to make a discord bot, and when I try to load a .env with load_dotenv() it doesn't work because it says Traceback (most recent call last): File "/home/fanjin/Documents/Python Projects/Discord Bot/bot.py", line 15, in <module> client.run(TOKEN) File "/home/fanjin/.local/lib/python3.8/site-packages/discord/client.py", line 708, in run return future.result() File "/home/fanjin/.local/lib/python3.8/site-packages/discord/client.py", line 687, in runner await self.start(*args, **kwargs) File "/home/fanjin/.local/lib/python3.8/site-packages/discord/client.py", line 650, in start await self.login(*args, bot=bot) File "/home/fanjin/.local/lib/python3.8/site-packages/discord/client.py", line 499, in login await self.http.static_login(token.strip(), bot=bot) AttributeError: 'NoneType' object has no attribute 'strip  Here's my code for the bot: import os import discord from dotenv import load_dotenv load_dotenv() TOKEN = os.getenv('DISCORD_TOKEN') client = discord.Client() @client.event async def on_ready(): print(f'{client.user} has connected to Discord!') client.run(TOKEN)  And the save.env file: (It's a fake token) # .env DISCORD_TOKEN={XXXXXXXX}  Both files are in the same directory, and I even tried to explicitly specify the .env's path with env_path = Path('path/to/file') / '.env' load_dotenv(dotenv_path=env_path)  but that also didn't work

Answer: TL:DR You need to put the full path. Use  either os.path.expanduser('~/Documents/MY_PROJECT/.env')  or: load_dotenv('/home/MY_USER/Documents/MY_PROJECT/.env')   and it will work. Or you change your current working directory in your code editor to where the ".env" file is (which should be the project folder). Or you open the project folder in the menu of your code editor, this should make the project folder the current working directory. On Linux, you can also go to the project folder in the terminal and start the code editor from there, type for example codium or whatever you use in the command prompt. Background Quote from the other answer  Since, I moved my .env file's inside another subfolder config, then I had to provide the full path to load_dotenv() to make it work.  This gave me the idea of checking the working directory. Current working directory os.getcwd() gave me a folder further up the tree. And then I copied the ".env" file into that working directory and it worked. Changing the working directory depends on your code editor. I use codium, which is the open source version of vscode, then you may follow for example Python in VSCode: Set working directory to python file's path everytime Full path You can also put the full path. Funny enough, I had checked that before coming here, but I copied the path that you get from the terminal, starting with '~/Documents/MY_PROJECT, which does not find the file but does not alert either, any tried environment variables were just empty - just because the ".env" file itself was never read.
Question: When operating with a venv on Windoes 10 if I invoke a subprocess from a file in the directory, the subprocess does not seem to have access to the venv. Is there a way to make it work? Ideally I would like the approach to be portable to Linux but I'll take whatever gets the project running. Here is my test:  main.py uses Popen to invoke sub_proc.py. sub_proc.py imports uuid_shortener, which has been installed in the virtual environment.  If I run sub_proc.py directly it runs without an error. However, if I run main.py I see an error on the import statement for uuid_shortener. main.py import subprocess import time print(subprocess.Popen(['python', 'sub_proc.py'])) time.sleep(1)  sub_proc.py import uuid_shortener  Here is the output from running the code. (venv) PS C:\Users\...\popenvenv> python .\sub_proc.py  (no error above) (venv) PS C:\Users\...\popenvenv> python .\main.py <Popen: returncode: None args: ['python', 'sub_proc.py']> Traceback (most recent call last): File "C:\Users\...\popenvenv\sub_proc.py", line 1, in <module> import uuid_shortener ModuleNotFoundError: No module named 'uuid_shortener' (venv) PS C:\Users\...\popenvenv> 

Answer: Use sys.executable in place of 'python'. sys.executable refers to the executable you're running with. This will preserve access to the virtualenv in subprocesses.
Question: I've seen the use of pyautogui.typewrite() but I can't find the documentation for it in the pyautogui docs. I did find the pyautogui.write() function in the docs and I wanted to know if they are the same thing, because from what I can see, they seem very similar.

Answer: As of version 1.0 there is no difference between write and typewrite. The write function has been chosen as the preferred invocation but currently write is just an alias for typewrite: write = typewrite # In PyAutoGUI 1.0, write() replaces typewrite().
Question: I am testing a class that needs a mock in the constructor, so I usually do this: class TestActionManager(unittest.TestCase): @patch('actionlib.SimpleActionClient', return_value=create_autospec(actionlib.SimpleActionClient)) def setUp(self, mock1): self.action_manager = ActionManager()  Then in this class I add all the tests. So the first one is working fine  def test_1(self): self.action_manager.f() self.action_manager.f.assert_called_once()  But if I add another test and run both  def test_2(self): self.action_manager.f() self.action_manager.f.assert_called_once()  It says f has been called twice. I was expecting setUp to create a new ActionManager (and hence create a new mock) before starting every test, but it is clearly not happening, since the mock is somehow shared. Also I tried to do def tearDown(self): del self.action_manager  But it does not fix the problem. I have read something related in Python Testing - Reset all mocks? where the solution is to use a different library (something that I would like to avoid) and in Any way to reset a mocked method to its original state? - Python Mock - mock 1.0b1 where it is using different classes to do it. Is there any possibility to reset the mock in the same class before or after every test?

Answer: I believe what you're looking for is reset_mock Here's, in general, how it works: def test_1(self): f = MagicMock() # or whatever you're mocking f() f.assert_called_once() f.reset_mock() f() f.assert_called_once()  The result will be PASSED If you want to automate, then you store the mocked thing inside setUp, and in tearDown you call the mocked thing's .reset_mock() method. def setUp(self, mock1): self.mock1 = mock1 # ... do other things ... def tearDown(self): self.mock1.reset_mock()