eshangj/stackoverflow_q_and_a_sample huggingface dataset:
Question: When I write a code in the editor in VScode and then I try to import this code into jupyter notebook, the alterations I made in the code are do not update - the code that runs in jupyter notebook is the code that is open when I initialize VScode. To update the code I need to restart VScode, open jupyter notebook, and import the code with the alterations made before restarting VScode. I set the path to the folder, and I using the same virtual environment. Is there anyway to fix this? 

Answer: You could download script file from jupyter as "py" file and run it on other editors most jupyter support that for example in anaconda jupyter  and google colab support this too but a notebook run only with program support python notebook.
Question: I prefer to use this extension so I find very annoying that every time I write a code-cell or hit shift-enter VS-Code opens its internal interactive python console. How to stop such a behavior?

Answer: You can view your keyboard shortcuts using:  Click the gear icon in the lower left corner of the interface  Select Keyboard Shortcuts (Ctrl + K + S)   Select Record Keys (Alt + k) on the right side of the input box   Press the SHIFT and ENTER keys  View the functions bound to this shortcut key in the list   Just delete what you don't want.     UPDATE Uninstall Jupyter extension, you will not get any interactive window features.  Don't forget to restart vscode after uninstalling
Question: I'm currently migrating some Python code that used to be blocking to use asyncio with async/await. It is a lot of code to migrate at once so I would prefer to do it gradually and have metrics. With that thing in mind I want to create a decorator to wrap some functions and know how long they are blocking the event loop. For example: def measure_blocking_code(f): def wrapper(*args, **kwargs): # ????? # It should measure JUST 1 second # not 5 which is what the whole async function takes return wrapper @measure_blocking_code async def my_function(): my_blocking_function() # Takes 1 seconds await my_async_function() # Takes 2 seconds await my_async_function_2() # Takes 2 seconds  I know the event loop has a debug function that already report this, but I need to get that information for specific functions.

Answer: TLDR; This decorator does the job: def measure_blocking_code(f): async def wrapper(*args, **kwargs): t = 0 coro = f() try: while True: t0 = time.perf_counter() future = coro.send(None) t1 = time.perf_counter() t += t1 - t0 while not future.done(): await asyncio.sleep(0) future.result() # raises exceptions if any except StopIteration as e: print(f'Function took {t:.2e} sec') return e.value return wrapper  Explanation This workaround exploits the conventions used in asyncio implementation in cPython. These conventions are a superset of PEP-492. In other words:  You can generally use async/await without knowing these details. This might not work with other async libraries like trio.  An asyncio coro object (coro) can be executed by calling .send() member. This will only run the blocking code, until an async call yields a Future object. By only measuring the time spent in .send(), the duration of the blocking code can be determined.
Question: I'm using python3 and would like to turn strings that contain all caps words (separately or inside a word) into title case (first letter capitalized). I do not want to disrupt one-off capital letters in the middle of a word (camel case), but if there are repeated capitalized letters, I want to keep only the first one capitalized. Here's the desired behavior >>> a = "TITLE BY DeSoto theHUMUNGUSone" >>> print(myfunc(a)) Title By DeSoto TheHumungusone  In words, "capitalize the beginning of each word and then take any letter that follows a capital letter and make it lower case." The str.title() does the initial letters the way I want, but it makes all intra-word letters lower case, rather than just those following the first. I was playing with a regular expression approach that makes anything that follows an upper case letter lower case, but I kept getting every other letter capitalized.

Answer: A regular expression substitution with a lambda is the way to go: import re a = "TITLE BY DeSoto theHUMUNGUSone" print(re.sub('[A-Z]+', lambda x: x.group(0).title(), a))  Output: Title By DeSoto theHumungusone
Question: I was debugging a C++ program in the gdb debugger and tried to access the 5th element of vector which only contains 4 element. After trying it, this error was on the screen: (gdb) list main 1 #include <memory> 2 #include <vector> 3 4 int main(int argc, char *argv[]){ 5 6 7 std::vector<int> v_num = {1, 3, 5, 67}; 8 std::unique_ptr<std::vector<int>> p(&v_num); 9 10 (gdb) p v_num.at (5) Python Exception <class 'IndexError'> Vector index "5" should not be >= 4.: Error while executing Python code. (gdb)  I didn't expect to see a Python exception inside the the gdb. Can someone explain why I encountered such error? Does gdb uses python internally?

Answer:  Does gdb uses python internally?  Yes, it uses Python a lot to extend itself in many ways, see https://sourceware.org/gdb/onlinedocs/gdb/Python.html#Python. What you discovered is called Python Xmethods, see https://sourceware.org/gdb/onlinedocs/gdb/Xmethods-In-Python.html. Xmethods are used as a replacement of inlined or optimized out method defined in C++ source code. libstdc++ has a number of xmethods for standard containers. You can see them with info xmethod command. For std::vector there are 6 xmethods defined on my box:  libstdc++::vector size empty front back at operator[]  You can disable all xmethods with disable xmethod command. If you do it, GDB will unable to call at method: (gdb) p v_num.at (5) Cannot evaluate function -- may be inlined
Question: I'm trying to convert csv format to JSON, I googled I'm not getting the correct way to modify it to get the desired one. This is my code in python: import csv import json def csv_to_json(csvFilePath, jsonFilePath): jsonArray = [] #reading csv (encoding is important) with open(csvFilePath, encoding='utf-8') as csvf: #csv library function csvReader = csv.DictReader(csvf) #convert each csv row into python dictionary for column in csvReader: #add this python dictionary to json array jsonArray.append(column) #convertion with open(jsonFilePath, 'w', encoding='utf-8') as jsonf: jsonString = json.dumps(jsonArray, indent=4) jsonf.write(jsonString) csvFilePath='example.csv' jsonFilePath='output.json' csv_to_json(csvFilePath, jsonFilePath)  and this is my csv file format:  My actual JSON Output: [ { "Area": "IT", "Employee": "Carl", }, { "Area": "IT", "Employee": "Walter", }, { "Area": "Financial Resources", "Employee": "Jennifer", } ]  My desired JSON Output: [ { "Area": "IT", "Employee": ["Carl","Walter"], }, { "Area": "Financial Resources", "Employee": ["Jennifer"], } ]  Thank you in advance!

Answer: Something like this should work. def csv_to_json(csvFilePath, jsonFilePath): areas = {} with open(csvFilePath, encoding='utf-8') as csvf: csvReader = csv.DictReader(csvf) for column in csvReader: area, employee = column["Area"], column["Employee"] # split values if area in areas: # add all keys and values to one dictionary areas[area].append(employee) else: areas[area] = [employee] # convert dictionary to desired output format. jsonArray = [{"Area": k, "Employee": v} for k,v in areas.items()] with open(jsonFilePath, 'w', encoding='utf-8') as jsonf: jsonString = json.dumps(jsonArray, indent=4) jsonf.write(jsonString)
Question: I am looking to multiply element-wise two dataframes with matching indices, using a dictionary to map which columns to multiply together. I can only come up with convoluted ways to do it and I am sure there is a better way, really appreciate the help! thx! df1:     Index ABC DEF XYZ     01/01/2004 1 2 3   05/01/2004 4 7 2     df2:     Index Echo Epsilon     01/01/2004 5 10   05/01/2004 -1 -2     Dictionary d = {'ABC': 'Echo', 'DEF': 'Echo', 'XYZ': 'Epsilon'} Expected result:     Index ABC DEF XYZ     01/01/2004 5 10 30   05/01/2004 -4 -7 -4    

Answer: You can use: # only if not already the index df1 = df1.set_index('Index') df2 = df2.set_index('Index') df1.mul(df2[df1.columns.map(d)].set_axis(df1.columns, axis=1))  or: df1.mul(df2.loc[df1.index, df1.columns.map(d)].values)  output:  ABC DEF XYZ Index 01/01/2004 5 10 30 05/01/2004 -4 -7 -4
Question: I have a pandas dataframe like this:  col_name 0 2 1 3 2 1 3 0 4 5 5 4 6 3 7 3  that could be created with the code: import pandas as pd dataframe = pd.DataFrame( { 'col_name': [2, 3, 1, 0, 5, 4, 3, 3] } )  Now, I want to get the rows which have a value less than the values in all the n previous rows. So, for n=2 the output is the rows: 2, 3, 6. Additionally I don't want to use any for-loops in my code. Just a trick that comes to my mind is that we can count the number of rows in n previous rows having a value less than the current row and then check if the number equals to n. Have you any idea for the code with my trick or in any other ways? and also is there a way to write a similar code except that if we want to check if values in n next rows are less than the value of the current row?

Answer: Let us use rolling to calculate the min value in n previous rows then compare the min value with current row to create a boolean mask df[df['col_name'] < df['col_name'].shift().rolling(2, min_periods=1).min()]    col_name 2 1 3 0 6 3
Question: I have been using dash leaflet for creating my own dashboard with maps, and it has been great to be able to visualize things with an interactive map. However, there is one thing that I have been stumped on a while for how to deal with a click happening on a polygon or marker. To explain this, I have created a simple example below import geopandas as gpd import dash_leaflet as dl import dash_leaflet.express as dlx from dash import Dash, html, Output, Input import json location = gpd.GeoDataFrame(geometry=gpd.points_from_xy([-74.0060], [40.7128])) app = Dash() app.layout = html.Div( children=[ dl.Map( center=[39, -98], zoom=4, children=[ dl.TileLayer(), dl.GeoJSON(data=dlx.geojson_to_geobuf(json.loads(location.to_json())), format='geobuf', id='locations', zoomToBoundsOnClick=True)], style={'width': '100%', 'height': '80vh'}, id="map"), html.Div(id='counter')]) counter = 0 @app.callback(Output('counter', 'children'), Input('locations', 'click_feature')) def update_counter(feature): global counter counter += 1 return str(counter) if __name__ == "__main__": app.run_server(debug=True)  And here is what the dashboard looks like when you first load it  Below the map there is a div that contains the number of times the geojson has been clicked (and I realize that when initializing that the function gets called, but that isn't the focus of this problem). When you click on the marker the first time, the div gets updated and the number increases. However, if you were to try and click on the marker again, the div does not update and there is no increase in the number. What I have figured out is that an event will only be fired if you click on a different marker (you can add a coordinate to the geojson and click between the two markers to see for yourself). In the example, this is a counter, but I am trying to filter my data to the location clicked on every time. So my question is what do you have to do to have your data filter every time the geojson is clicked on? Another small thing that I have noticed is that even though I set zoomToBoundsOnClick to true, I do not have the map zoom in on the marker that I clicked on. This isn't a big deal, but it would be a nice to have. So if someone know how to get that work, that would also be appreciated.

Answer: In Dash, a callback is only invoked when a property changes. If you click the same feature twice, the click_feature property doesn't change, and the callback is thus not invoked. If you want to invoke the callback on every click, you can target the n_clicks property - it is incremented on (every) click, and the callback will thus fire on every click.
Question: I am new at programming (please be nice :)) and I am trying to write a function that calculates the binary gap of a number using recursion by using the modulus function to gather the bits of an integer and then counting the number of 0s between 1s and displaying the longest chain 0s in between two 1s. For example Input = 10, Binary value = 1010, Binary Gap = 1 Input = 15, Binary value = 1111, Binary Gap = 0 Input = 41, Binary value = 101001, Binary Gap = 2  As it stands the code below will always return a 0 if an even number is inputted and always throw a recursion error should an odd number be used. This leads me to believe there is a problem involving the division and it being stuck in an infinite loop but cannot figure out why. def binary_gap(n, foundFirstOne= False, numofZeros= 0): if n == 0: return 0 bit = n % 2 if bit == 1: ans = binary_gap((n / 2), True, 0) else: if foundFirstOne == True: ans = binary_gap((n / 2), True, numofZeros+1) else: ans = binary_gap((n / 2), False, 0) return ans and numofZeros print(binary_gap(int(input("Please enter a number")))) 

Answer: First, write a recursive function to convert decimal to binary (not necessary because there is already a bin function). Then, count the gap - there is no point using recursion in this because it is just way too complicated: def convert_to_binary(n): if n > 1: return convert_to_binary(n // 2) + str(n % 2) return str(n % 2) def binary_gap(n): b = convert_to_binary(n) x = b.split('1')[:-1] return max(len(d) for d in x) print(binary_gap(10)) # 1 print(binary_gap(15)) # 0 print(binary_gap(41)) # 2
Question: I am trying to type hint the arguments of a function that takes a callable, and has a default argument (in the example below set) from typing import Callable, List T = TypeVar("T") def transform(data: List[int], ret_type: Callable[[List[int]], T] = set) -> T: return ret_type(data) a = [1, 2, 3] my_set: Set = transform(a)  The above code triggers the following error message from mypy mypy3: Incompatible default for argument "ret_type" (default has type "Type[Set[Any]]", argument has type "Callable[[List[int]], T]") What should be the correct type of ret_type ? EDIT The below code although not ideal works fine (cf @chepner comment) from typing import cast, Any, Callable, TypeVar T = TypeVar("T") def transform(data: Any, ret_type: Callable[..., T] = cast(Callable, set)) -> T: return ret_type(data) 

Answer: You could use the @overload for correctly type hinting of function with default argument for your case: from typing import Callable, List, TypeVar, overload, Set T = TypeVar("T") @overload def transform(data: List[int]) -> Set[int]: ... @overload def transform(data: List[int], ret_type: Callable[[List[int]], T]) -> T: ... # untyped implementation def transform(data, ret_type = set): return ret_type(data) a = [1, 2, 3] my_set: Set = transform(a)
Question: What's the Pythonic way of checking whether a string is None, empty or has only whitespace (tabs, spaces, etc)? Right now I'm using the following bool check: s is None or not s.strip()  ..but was wondering if there's a more elegant / Pythonic way to perform the same check. It may seem easy but the following are the different issues I found with this:  isspace() returns False if the string is empty. A bool of string that has spaces is True in Python. We cannot call any method, such as isspace() or strip(), on a None object. 

Answer: The only difference I can see is doing: not s or not s.strip()  This has a little benefit over your original way that not s will short-circuit for both None and an empty string. Then not s.strip() will finish off for only spaces. Your s is None will only short-circuit for None obviously and then not s.strip() will check for empty or only spaces.
Question: I'm looking for an algorithm to create a new column based on values ​​from other columns AND respecting pre-established rules. Here's an example: artificial data df = data.frame( col_1 = c('No','Yes','Yes','Yes','Yes','Yes','No','No','No','Unknown'), col_2 = c('Yes','Yes','Unknown','Yes','Unknown','No','Unknown','No','Unknown','Unknown'), col_3 = c('Unknown','Yes','Yes','Unknown','Unknown','No','No','Unknown','Unknown','Unknown') )  The goal is to create a new_column based on the values ​​of col_1, col_2, and col_3. For that, the rules are:  If the value 'Yes' is present in any of the columns, the value of the new_column will be 'Yes'; If the value 'Yes' is not present in any of the columns, but the value 'No' is present, then the value of the new_column will be 'No'; If the values ​​'Yes' and 'No' are absent, then the value of new_columns will be 'Unknown'.  I managed to operationalize this using case_when() describing all possible combinations; or ifelse sequential. But these solutions are not scalable to N variables. Current solution: library(dplyr) df_1 <- df %>% mutate( new_column = ifelse( (col_1 == 'Yes' | col_2 == 'Yes' | col_3 == 'Yes'), 'Yes', ifelse( (col_1 == 'Unknown' & col_2 == 'Unknown' & col_3 == 'Unknown'), 'Unknown','No' ) ) )  I'm looking for some algorithm capable of operationalizing this faster and capable of being expanded to N variables. After searching for StackOverflow, I couldn't find a way to my problem (I know there are several posts about creating a new column based on values ​​obtained from different columns, but none). Perhaps the search strategy was not the best. If anyone finds it, please provide the link. I used R in the code, but the current solution works in Python using np.where. Solutions in R or Python are welcome.

Answer: A solution using Python: import pandas as pd df = pd.DataFrame({ 'col_1': ['No','Yes','Yes','Yes','Yes','Yes','No','No','No','Unknown'], 'col_2': ['Yes','Yes','Unknown','Yes','Unknown','No','Unknown','No','Unknown','Unknown'], 'col_3': ['Unknown','Yes','Yes','Unknown','Unknown','No','No','Unknown','Unknown','Unknown'] }) df['col_4'] = [('Yes' if 'Yes' in x else ('No' if 'No' in x else 'Unknown')) for x in zip(df['col_1'], df['col_2'], df['col_3'])] print(df)  Output:  col_1 col_2 col_3 col_4 0 No Yes Unknown Yes 1 Yes Yes Yes Yes 2 Yes Unknown Yes Yes 3 Yes Yes Unknown Yes 4 Yes Unknown Unknown Yes 5 Yes No No Yes 6 No Unknown No No 7 No No Unknown No 8 No Unknown Unknown No 9 Unknown Unknown Unknown Unknown
Question: I have the following Github action, in which I'm specifying Python 3.10: name: Unit Tests runs-on: ubuntu-latest defaults: run: shell: bash working-directory: app steps: - uses: actions/checkout@v3 - name: Install poetry run: pipx install poetry - uses: actions/setup-python@v3 with: python-version: "3.10" cache: "poetry" - run: poetry install - name: Run tests run: | make mypy make test  The pyproject.toml specifies Python 3.10 as well: [tool.poetry.dependencies] python = ">=3.10,<3.11"  When the action runs, I get the following: The currently activated Python version 3.8.10 is not supported by the project (>=3.10,<3.11). Trying to find and use a compatible version. Using python3 (3.10.5)  It would look like it's using 3.10, but py.test is using 3.8.10: platform linux -- Python 3.8.10, pytest-6.2.5, py-1.11.0, pluggy-1.0.0 -- /home/runner/.cache/pypoetry/virtualenvs/vital-background-pull-yluVa_Vi-py3.10/bin/python  For context, this Github action was running on 3.8 before. I've updated the python version in both the test.yaml and the pyproject.toml but it's still using 3.8. Anything else I should change to make it use 3.10? Thank you

Answer: The root cause is the section - uses: actions/setup-python@v3 with: python-version: "3.10" cache: "poetry"  with the line caching poetry. Since poetry was previously installed with a pip associated with Python 3.8, the package will be retrieved from the cache associated with that Python version. It needs to be re-installed with the new Python version. You can either remove the cache: poetry from a single GH actions execution, or remove the cache manually. This will fix your issue.
Question: I have fine-tuned a PyTorch transformer model using HuggingFace, and I'm trying to do inference on a GPU. However, even after setting model.eval() I still get slightly different outputs if I run inference multiple times on the same data. I have tried a number of things and have done some ablation analysis and found out that the only way to get deterministic output is by also setting torch.cuda.manual_seed_all(42)  (or any other seed number). Why is this the case? And is this normal? The model's weights are fixed, and there are no undefined or randomly initialized weights (when I load the trained model I get the All keys matched successfully message), so where is the randomness coming from if I don't set the cuda seed manually? Is this randomness to be expected?

Answer: You can use torch.use_deterministic_algorithms to force non-deterministic modules to perform deterministically, where supported e.g: >>> a = torch.randn(100, 100, 100, device='cuda').to_sparse() >>> b = torch.randn(100, 100, 100, device='cuda') # Sparse-dense CUDA bmm is usually nondeterministic >>> torch.bmm(a, b).eq(torch.bmm(a, b)).all().item() False >>> torch.use_deterministic_algorithms(True) # Now torch.bmm gives the same result each time, but with reduced performance >>> torch.bmm(a, b).eq(torch.bmm(a, b)).all().item() True # CUDA kthvalue has no deterministic algorithm, so it throws a runtime error >>> torch.zeros(10000, device='cuda').kthvalue(1) RuntimeError: kthvalue CUDA does not have a deterministic implementation...
Question: Python computes the imaginary unit i = sqrt(-1) inaccurately: >>> (-1) ** 0.5 (6.123233995736766e-17+1j)  Should be exactly 1j (Python calls it j instead of i). Both -1 and 0.5 are represented exactly, the result can be represented exactly as well, so there's no hard reason (i.e., floating point limitations) why Python couldn't get it right. It could. And i=sqrt(-1) being the definition makes it rather disappointing that Python gets that wrong. So why does it? How does it compute the inaccurate result?

Answer: When complex arithmetic is required, your Python implementation likely calculates xy as ey ln x, as might be done with the complex C functions cexp and clog. Those are in turn likely calculated with real functions including ln, sqrt, atan2, sin, cos, and pow, but the details need not concern us. ln −1 is πi. However, π is not representable in a floating-point format. Your Python implementation likely uses the IEEE-754 “double precision” format, also called binary64. In that format, the closest representable value to π is 3.141592653589793115997963468544185161590576171875. So ln −1 is likely calculated as 3.141592653589793115997963468544185161590576171875 i. Then y ln x = .5 • 3.141592653589793115997963468544185161590576171875 i is 1.5707963267948965579989817342720925807952880859375 i. e1.5707963267948965579989817342720925807952880859375 i is also not exactly representable. The true value of that is approximately 6.123233995736765886130329661375001464640•10−17 + .9999999999999999999999999999999981253003 i. The nearest representable value to 6.123233995736765886130329661375001464640•10−17 is 6.12323399573676603586882014729198302312846062338790031898128063403419218957424163818359375•10−17, and the nearest representable value to .9999999999999999999999999999999981253003 is 1, so the calculated result is 6.12323399573676603586882014729198302312846062338790031898128063403419218957424163818359375•10−17 + 1 i.