eshangj/stackoverflow_q_and_a_sample huggingface dataset:
Question: What is the equivalent of header=0 in pandas, which recognises the first line as a heading in gspread? pandas import statement (correct) import pandas as pd # gcp / google sheets URL df_URL = "https://docs.google.com/spreadsheets/d/1wKtvNfWSjPNC1fNmTfUHm7sXiaPyOZMchjzQBt1y_f8/edit?usp=sharing" raw_dataset = pd.read_csv(df_URL, na_values='?',sep=';' , skipinitialspace=True, header=0, index_col=None)  Using the gspread function, so far I import the data, change the first line to the heading then delete the first line after but this recognises everything in the DataFrame as a string. I would like to recognise the first line as a heading right away in the import statement. gspread import statement that needs header=True equivalent import pandas as pd from google.colab import auth auth.authenticate_user() import gspread from oauth2client.client import GoogleCredentials # gcp / google sheets url df_URL = "https://docs.google.com/spreadsheets/d/1wKtvNfWSjPNC1fNmTfUHm7sXiaPyOZMchjzQBt1y_f8/edit?usp=sharing" # importing the data from Google Drive setup gc = gspread.authorize(GoogleCredentials.get_application_default()) # read data and put it in dataframe g_sheets = gc.open_by_url(df_URL) df = pd.DataFrame(g_sheets.get_worksheet(0).get_all_values()) # change first row to header df = df.rename(columns=df.iloc[0]) # drop first row df.drop(index=df.index[0], axis=0, inplace=True) 

Answer: Looking at the API documentation, you probably want to use: df = pd.DataFrame(g_sheets.get_worksheet(0).get_all_records(head=1))  The .get_all_records method returns a dictionary of with the column headers as the keys and a list of column values as the dictionary values. The argument head=<int> determines which row to use as keys; rows start from 1 and follow the numeration of the spreadsheet. Since the values returned by .get_all_records() are lists of strings, the data frame constructor, pd.DataFrame, will return a data frame that is all strings. To convert it to floats, we need to replace the empty strings, and the the dash-only strings ('-') with NA-type values, then convert to float. Luckily pandas DataFrame has a convenient method for replacing values .replace. We can pass it mapping from the string we want as NAs to None, which gets converted to NaN. import pandas as pd data = g_sheets.get_worksheet(0).get_all_records(head=1) na_strings_map= { '-': None, '': None } df = pd.DataFrame(data).replace(na_strings_map).astype(float)
Question: When uploading to PyPI there is an error: $ twine upload -r test dist/examplepkg-1.0.tar.gz Uploading distributions to https://test.pypi.org/legacy/ Uploading examplepkg-1.0.tar.gz Error during upload. Retry with the --verbose option for more details. HTTPError: 400 Bad Request from https://test.pypi.org/legacy/ The name 'examplepkg' is too similar to an existing project. See https://test.pypi.org/help/#project-name for more information.  Which existing project? How do you find out what existing project is it talking about?

Answer: There is no direct way of knowing which exact package causes the name conflict, but here are some tips that may help you further in your search. First of all, you can find the source code of pypi (called warehouse) at https://github.com/pypa/warehouse/. Using the error message you gave, you can find that the failing check is caused by a database function called ultranormalize_name. Now, searching for that name in the codebase leads you to this migration script where the function seems to be created, which performs the following steps to check if the name is already reserved:  Both cases of o (lower and upper case, o and O) gets replaced with 0 (irrelevant for your case, as there are no os in your package name) Both cases of L and I are replaced with 1 (e.g., example is same as examp1e and exampie) All ., _, and - characters are removed (e.g., e-x-a-m-p-l-e is same as example) The result is then lowercased and compared to the already existing names  As I cannot see a direct match for your given package name, are you sure examplepkg is the name that is also in your pyproject.toml or setup.py file metadata? If yes, there is probably some variant of that name whose non-normalized form matches to yours after the transformations mentioned above.
Question: I have the following string: seq = 'MNRYLNRQRLYNMYRNKYRGVMEPMSRMTMDFQGRYMDSQGRMVDPRYYDHYGRMHDYDRYYGRSMFNQGHSMDSQRYGGWMDNPERYMDMSGYQMDMQGRWMDAQGRYNNPFSQMWHSRQGH'  also saved in a file called seq.dat. If I use the following grep command grep '\([MF]D.\{4,6\}\)\{3,10\}' seq.dat  I get the following matching string: MDNPERYMDMSGYQMDMQGRWMDAQGRYN  which is what I want. In words, what I want to match is as many consecutive repeats as the string has of [MF]D.{4,6}. I don't want to match cases where it has less than 3 consecutive repeats, but I want it to be able to capture up to 6. Now, I'm trying to do this with python. I have p = re.compile("(?:[MF]D.{4,6}){3,10}")  Trying search() returns MDNPERYMDMSGYQMDMQGRWM  It is the close to the answer I seek, but is still missing the last MDAQGRYN. I'm guessing this is because .{4,6} matches the M, which in turn prevents {3,10} from capturing this 4th occurence of ([MF]D.{4,6}), but since I asked for at least 3, it's happy and it stops. How do I make Python regex behave like grep does?

Answer: There is a fundamental difference between POSIX ("text-directed") and NFA ("regex-directed") engines. POSIX engines (grep here uses a POSIX BRE regex flavor, it is the flavor used by default) will parse the input text applying the regex to it and return the longest match possible. NFA engine (Python re engine is an NFA engine) here does not re-consume (backtrack) when the subsequent pattern parts match. See reference on regex-directed and text-directed engines:  A regex-directed engine walks through the regex, attempting to match the next token in the regex to the next character. If a match is found, the engine advances through the regex and the subject string. If a token fails to match, the engine backtracks to a previous position in the regex and the subject string where it can try a different path through the regex... Modern regex flavors using regex-directed engines have lots of features such as atomic grouping and possessive quantifiers that allow you to control this backtracking. A text-directed engine walks through the subject string, attempting all permutations of the regex before advancing to the next character in the string. A text-directed engine never backtracks. Thus, there isn’t much to discuss about the matching process of a text-directed engine. In most cases, a text-directed engine finds the same matches as a regex-directed engine.  The last sentence says "in most cases", but not all cases, and yours is a good illustration that discrepances may occur. To avoid consuming M or F that are immediately followed with D, I'd suggest using (?:[MF]D(?:(?![MF]D).){4,6}){3,10}  See the regex demo. Details:  (?: - start of an outer non-capturing container group:  [MF]D - M or F and then D (?:(?![MF]D).){4,6} - any char (other than a line break) repeated four to six times, that does not start an MD or FD char sequence   ){3,10} - end of the outer group, repeat 3 to 10 times.  By the way, if you only want to match uppercase ASCII letters, replace the . with [A-Z].
Question: So for some reason my intellisense is not working. I tried the solutions suggested here Visual Studio Code: Intellisense not working. The solution that seems to help most people is adding "python.autoComplete.extraPaths": [ "${workspaceFolder}/customModule" ], didn't work. Also VS Code says it doesn't recognize python.pythonPath when I add it. Auto-complete not working, screen capture didn't capture my cursor, but it's right after argparse., which should give the option to auto-complete with a list that includes: ArgumentParser:  Remote server installed extensions:  Settings.json This is settings.json on remote server { "remote.autoForwardPortsSource": "output", "python.languageServer": "None", "python.analysis.completeFunctionParens": true, "python.analysis.diagnosticMode": "workspace", }  Setup:  Running using Conda env Linux remote server  Note: Something else off is my "find declaration of function or class" is also not working.

Answer: the first solutions are kind of obvious, but ill add them anyway,  Removing reinstalling it both locally and remotely Make sure VS code is updated to its last version In settings.json, set a language server in "python.languageServer". The Language Server includes: Jedi(build-in Python extension ), Microsoft, Pylance, since you have already installed Pylance, let's start with that one (if that doesnt work, try the others). set your python.pythonPath to the path returned in your terminal for which python3
Question: When running my code I get the below error string, <string>:36: DeprecationWarning: executable_path has been deprecated, please pass in a Service object  What could possibly be the issue? Below is the Selenium setup, options = webdriver.ChromeOptions() prefs = {"download.default_directory" : wd} options.add_experimental_option("prefs", prefs) options.add_argument("--headless") path = (chrome) driver = webdriver.Chrome(executable_path=path, options = options) driver.get('https://www.1linelogin.williams.com/1Line/xhtml/login.jsf?BUID=80') 

Answer: This error message  DeprecationWarning: executable_path has been deprecated, please pass in a Service object  means that the key executable_path will be deprecated in the upcoming releases. Once the key executable_path is deprecated you have to use an instance of the Service() class as follows: from selenium import webdriver from selenium.webdriver.chrome.service import Service path = (chrome) s = Service(path) driver = webdriver.Chrome(service=s)  For more details see here
Question: Is there a scikit-learn preprocesser I can use or implement to select a subset of rows from a pandas dataframe? I would prefer a preprocesser to do this since I want to build a pipeline with this as a step.

Answer: You can use a FunctionTransformer to do that. To a FunctionTransformer, you can pass any Callable that exposes the same interface as standard scikitlearn transform calls have. In code import pandas as pd from sklearn.preprocessing import FunctionTransformer class RowSelector: def __init__(self, rows:list[int]): self._rows = rows def __call__(self, X:pd.DataFrame, y=None) -> pd.DataFrame: return X.iloc[self._rows,:] selector = FunctionTransformer(RowSelector(rows=[1,3])) df = pd.DataFrame({'a':range(4), 'b':range(4), 'c':range(4)}) selector.fit_transform(df) #Returns a b c 1 1 1 1 3 3 3 3  Not that, I have used a callable object to track some state, i.e. the rows to be selected. This is not necessary and could be solved differently. The cool thing is that it returns a data frame, so if you have it as the first step of your pipeline, you can also combine it with a subsequent column transformer (if needed of course)
Question: What I'm trying to do Test a website that requires microphone access with playwright The problem Pop-up in question comes up and seems to ignore supposedly granted permissions. Permission can be given manually, but this seems against the spirit of automation.  What I tried with sync_playwright() as p: browser = p.chromium.launch(headless=False) context = browser.new_context(permissions=['microphone']) ...  Granting permissions via context doesn't work for some reason. The permission pop-up still comes up. I also tried to record a walkthrough with playwrights record script, but it's not recording granting microphone permissions.

Answer: You're missing some command line flags that tell chrome to simulate having a microphone. Give this sample a shot. from playwright.sync_api import sync_playwright def run(playwright): chromium = playwright.chromium browser = chromium.launch(headless=False, args=['--use-fake-device-for-media-stream', '--use-fake-ui-for-media-stream']) context = browser.new_context() context.grant_permissions(permissions=['microphone']) page = context.new_page() page.goto("https://permission.site/") page.click('#microphone') page.pause() # other actions... browser.close() with sync_playwright() as playwright: run(playwright)
Question: what would be a recommended way to install your Python's package dependencies with poetry for Azure Pipelines? I see people only downloading poetry through pip which is a big no-no. - script: | python -m pip install -U pip pip install poetry poetry install displayName: Install dependencies  I can use curl to download poetry.  - script: | curl -sSL https://raw.githubusercontent.com/python-poetry/poetry/master/get-poetry.py | python - export PATH=$PATH:$HOME/.poetry/bin poetry install --no-root displayName: 'Install dependencies'  But then in each subsequent step I have to add poetry to PATH again ...  - script: | curl -sSL https://raw.githubusercontent.com/python-poetry/poetry/master/get-poetry.py | python - export PATH=$PATH:$HOME/.poetry/bin poetry install --no-root displayName: 'Install dependencies' - script: | # export PATH=$PATH:$HOME/.poetry/bin poetry run flake8 src displayName: 'Linter' - script: | # export PATH=$PATH:$HOME/.poetry/bin poetry add pytest-azurepipelines poetry run pytest src displayName: 'Tests'  Is there any right way to use poetry in Azure Pipelines?

Answer: Consulted this issue with a collegue. He recommended doing separate step to add Poetry to the PATH.  - task: UsePythonVersion@0 inputs: versionSpec: '3.8' displayName: 'Use Python 3.8' - script: | curl -sSL https://raw.githubusercontent.com/python-poetry/poetry/master/get-poetry.py | python - export PATH=$PATH:$HOME/.poetry/bin poetry install --no-root displayName: 'Install dependencies' - script: echo "##vso[task.prependpath]$HOME/.poetry/bin" displayName: Add poetry to PATH - script: | poetry run flake8 src displayName: 'Linter' - script: | poetry add pytest-azurepipelines poetry run pytest src displayName: 'Tests'
Question: #Segmenting the red pointer img = cv2.imread('flatmap.jpg') hsv = cv2.cvtColor(img, cv2.COLOR_BGR2HSV) lower_red = np.array([140, 110, 0]) upper_red = np.array([255, 255 , 255]) # Threshold with inRange() get only specific colors mask_red = cv2.inRange(hsv, lower_red, upper_red) # Perform bitwise operation with the masks and original image red_pointer = cv2.bitwise_and(img,img, mask= mask_red) # Display results cv2.imshow('Red pointer', red_pointer) cv2.imwrite('redpointer.jpg', red_pointer) cv2.waitKey(0) cv2.destroyAllWindows()  I have a map and need to extract the red arrow. The code works but the arrow has black patches in it. How would I go about altering the code to improve the output of the arrow so it's a solid shape?  

Answer: I've looked at the channels in HSL/HSV space. The arrows are the only stuff in the picture that has any saturation. That would be one required (but insufficient) aspect to get a lock on the desired arrow. I've picked those pixels and they appear to have a bit more than 50% saturation, so I'll use a lower bound of 25% (64).  That red arrow's hue dithers around 0 degrees (red)... that means some of its pixels are on the negative side of 0, i.e. something like 359 degrees.  You need to use two inRange calls to collect all hues from 0 up, and all hues from 359 down. Since OpenCV encodes hues in 2-degree steps, that'll be a value of 180 and down. I'll select 0 +- 20 degrees (0 .. 10 and 170 .. 180). In summary: hsv_im = cv.cvtColor(im, cv.COLOR_BGR2HSV) mask1 = cv.inRange(hsv_im, np.array([0, 64, 0]), np.array([10, 255, 255])) mask2 = cv.inRange(hsv_im, np.array([170, 64, 0]), np.array([180, 255, 255])) mask = mask1 | mask2 cv.imshow("mask", mask) cv.waitKey()
Question: I am trying to deploy a Python project to a machine with no internet. Because it has no internet, I cannot pip install any packages with a requirements.txt file. I am wondering if it is possible to move my existing environment with all installed packages into another machine with all packages pre-installed. I can also use attempt to use Docker for this installation. Would I be able to pre-install all the packages within a Docker container and then copy all the files onto another VM?

Answer: On you local machine (adapt the instructions if you are on Windows)  Create your requirements.txt file  (venv) [...]$ mkdir pkgs (venv) [...]$ cd pkgs (venv) [...]$ pip freeze > requirements.txt (venv) [...]$ pip download -r requirements.txt   Download pip archive from here  Copy pkgs folder to the remote machine   On the remote machine:  Install pip from archive  (venv) [...]$ cd pkgs # --- unarchive pip.tar.gz --- (venv) [...]$ python setup.py install   Install packages  (venv) [...]$ pip install --no-index --find-links . -r requirements.txt
Question: I have the string banana | 10 and want to replace everything from |  to the end of the string with 9. The output I want would be banana | 9. How could I achieve this? I've looked into .replace(), .split() and converting the string into a list of characters, and looping over them until I find the bit that should be replaced, but just couldn't figure it out.

Answer: I suggest you use re module(regex module): import re myString = "banana | 10" re.sub(r"\|.+", r"| 9", myString)  Output banana | 9
Question: I have a JSON with the following structure below into a list in a python variable. I'd like to extract this JSON value as a table. My question is, how can I extract it from the list and how can I change it into a table? Once I have converted it, I will insert the output into a Postgres table. JSON structure [' { "_id": { "$Col1": "XXXXXXX2443" }, "col2": false, "col3": "359335050111111", "startedAt": { "$date": 1633309625000 }, "endedAt": { "$date": 1633310213000 }, "col4": "YYYYYYYYYYYYYYYYYY", "created_at": { "$date": 1633310846935 }, "updated_at": { "$date": 1633310846935 }, "__v": 0 } ']  Desired output: 

Answer: Use the code below. I have used PrettyTable module for printing in a table like structure. Use this - https://www.geeksforgeeks.org/how-to-make-a-table-in-python/ for table procedure. Also, all the headers and values will be stored in headers and values variable. import json from prettytable import PrettyTable value = [''' { "_id": { "$Col1": "XXXXXXX2443" }, "col2": false, "col3": "359335050111111", "startedAt": { "$date": 1633309625000 }, "endedAt": { "$date": 1633310213000 }, "col4": "YYYYYYYYYYYYYYYYYY", "created_at": { "$date": 1633310846935 }, "updated_at": { "$date": 1633310846935 }, "__v": 0 }'''] dictionary = json.loads(value[0]) headers = [] values = [] for key in dictionary: head = key value = "" if type(dictionary[key]) == type({}): for key2 in dictionary[key]: head += "/" + key2 value = dictionary[key][key2] headers.append(head) values.append(value) else: value = dictionary[key] headers.append(head) values.append(value) print(headers) print(values) myTable = PrettyTable(headers) myTable.add_row(values) print(myTable)  Output ['_id/$Col1', 'col2', 'col3', 'startedAt/$date', 'endedAt/$date', 'col4', 'created_at/$date', 'updated_at/$date', '__v'] ['XXXXXXX2443', False, '359335050111111', 1633309625000, 1633310213000, 'YYYYYYYYYYYYYYYYYY', 1633310846935, 1633310846935, 0] +-------------+-------+-----------------+-----------------+---------------+--------------------+------------------+------------------+-----+ | _id/$Col1 | col2 | col3 | startedAt/$date | endedAt/$date | col4 | created_at/$date | updated_at/$date | __v | +-------------+-------+-----------------+-----------------+---------------+--------------------+------------------+------------------+-----+ | XXXXXXX2443 | False | 359335050111111 | 1633309625000 | 1633310213000 | YYYYYYYYYYYYYYYYYY | 1633310846935 | 1633310846935 | 0 | +-------------+-------+-----------------+-----------------+---------------+--------------------+------------------+------------------+-----+
Question: I have a simple code here: import mss with mss.mss() as sct: filename = sct.shot(output="result.png")  result.png  But I want to take a part of screen like this:  Thanks for help!

Answer: As explained on https://python-mss.readthedocs.io/examples.html, something like this should work: with mss.mss() as sct: # The screen part to capture monitor = {"top": 160, "left": 160, "width": 160, "height": 135} output = "sct-{top}x{left}_{width}x{height}.png".format(**monitor) # Grab the data sct_img = sct.grab(monitor) # Save to the picture file mss.tools.to_png(sct_img.rgb, sct_img.size, output=output) print(output)  This is just the example given on the website. You can adjust the part of the screen that you are taking a screenshot of by modifying the monitor dictionary. As an example you could change it from {"top": 160, "left": 160, "width": 160, "height": 135} to {"top": 10, "left": 14, "width": 13, "height": 105}. You will have to modify it to capture the part of the screen that you want.
Question: I'm working with an API where the schema for creating a group is effectively: class Group(BaseModel): identifier: str  I was hoping I could do this instead: class Group(BaseModel): groupname: str = Field(..., alias='identifier')  But with that configuration it's not possible to set the attribute value using the name groupname. That is, running this fails with a field required error: >>> g = Group(groupname='foo') pydantic.error_wrappers.ValidationError: 1 validation error for Group identifier field required (type=value_error.missing)  Is it is possible to use either the alias or the actual attribute name to set the attribute value? I was hoping that these two would be equivalent: >>> Group(identifier='foo') >>> Group(groupname='foo') 

Answer: Maybe you are looking for the allow_population_by_field_name config option:  whether an aliased field may be populated by its name as given by the model attribute, as well as the alias (default: False)  from pydantic import BaseModel, Field class Group(BaseModel): groupname: str = Field(..., alias='identifier') class Config: allow_population_by_field_name = True print(repr(Group(identifier='foo'))) print(repr(Group(groupname='bar')))  Output: Group(groupname='foo') Group(groupname='bar')