eshangj/stackoverflow_q_and_a_sample huggingface dataset:
Question: def my_func(): t = 10 while(t > 0): t = t - 1 for item in range(10): pass  Is there some easy way in Python to know that two loop constructs are getting used in the above code? I am trying to build a coding platform where I want to restrict users to using just one loop to solve a question, so this information can be precious for checking and restricting the users.

Answer: As others mentioned, you could use ast. code = """ def my_func(): t = 10 while(t > 0): t = t - 1 for item in range(10): pass """  import ast from typing import List, Union class LoopVisitor(ast.NodeVisitor): def __init__(self): self._loops: List[Union[ast.While, ast.For]] = [] @property def count(self) -> int: return len(self._loops) @property def loops(self) -> List[Union[ast.While, ast.For]]: return self._loops def visit_While(self, node: ast.While) -> None: self._loops.append(node) self.generic_visit(node) def visit_For(self, node: ast.For) -> None: self._loops.append(node) self.generic_visit(node) tree = ast.parse(code) visitor = LoopVisitor() visitor.visit(tree) print(visitor.count)  More information about ast.NodeVisitor here and here.  Alternatively you could also use tokenize. However, it will only work with syntactically valid Python code. import io import tokenize def count_loops(code: str) -> int: count = 0 for token in tokenize.tokenize(io.BytesIO(code.encode("utf-8")).readline): if token.type == tokenize.NAME and token.string in ("for", "while"): count += 1 return count print(count_loops(code))  More information about tokenize here and here  If you want to use a third-party library, you could also investigate parso using parso.python.tokenize.tokenize: from parso.python import tokenize def count_loops(code: str) -> int: count = 0 for token in tokenize.tokenize(code, version_info=(3, 9, 0)): if token.type == tokenize.NAME and token.string in ("for", "while"): count += 1 return count print(count_loops(code))  Or, by using parso.parse, however, please note that I only tested the example against the code example in your question. It may not work universally. from parso import parse from parso.tree import BaseNode from parso.python.tree import ForStmt, WhileStmt node: BaseNode = parse(code, version="3.9") def count_loops(node: BaseNode) -> int: count = 0 for child in node.children: if isinstance(child, (ForStmt, WhileStmt)): count += 1 elif isinstance(child, BaseNode): count += count_loops(child) return count print(count_loops(node))   Here is also a solution using asttokens: import asttokens def count_loops(code: str) -> int: count = 0 for token in asttokens.ASTTokens(code).tokens: if token.string in ("for", "while"): count += 1 return count print(count_loops(code))
Question: More often than not, I make the following mistake: import enum class StatusValues(enum.Enum): one = "one" two = "two" def status_is_one(status: str): return status == StatusValues.one  String will never be an enum class. The problem is that it should be StatusValues.one.value. Is there a strcmp(status, StatusValues.one)-ish function so that my pyright will error on the line that I am comparing string with a class? Is there a good way to protect against such mistakes?

Answer: Yes, this can be a gotcha: by default we cannot directly compare strings with enum members: >>> status = "two" >>> status == StatusValues.two False  One has to remember to compare a string to the enum member's .value, which is also kind of verbose: >>> "two" == StatusValues.two.value True  Fortunately, there is a solution, which is even mentioned in the docs, but it used to be (pre- Python 3.11) in a section that was somewhat easy to miss (IMHO). We can mix in a type of enum members: class IntEnum(int, Enum): pass   This demonstrates how similar derived enumerations can be defined; for example a StrEnum that mixes in str instead of int.  The solution is therefore to define an enum with a str base class: class StatusValues(str, Enum): one = "one" two = "two" >>> status = "two" >>> status == StatusValues.two # no .value True  In Python 3.11+ StrEnum is already included in the enum module, but you mentioned that you still need to support Python 3.7.
Question: As it's usually advised, I have managed to reduce my problem to a minimal reproducible example: import numpy as np import seaborn as sns import matplotlib.pyplot as plt matrix = np.array([[0.1234, 1.4567, 0.7890, 0.1234], [0.9876, 0, 0.5432, 0.6789], [0.1111, 0.2222, 0, 0.3333], [0.4444, 0.5555, 0.6666, 0]]) sns.heatmap(matrix, annot=True) plt.show()  Vaguely based on Seaborn official documentation. Unfortunately, unlike what would be expected (all numbers visible), I get only the numbers in the top row visible:   As there is not really much room for error in this one, I'm out of ideas and google/SO doesn't seem to have this question asked before. Is this a bug?  I am running: Seaborn 0.12.2 Matplotlib 3.8.0 PyCharm 2023.1.4 Windows 10 

Answer: Just ran into the issue myself, I was on Seaborn 0.12.2. Ran pip install seaborn --upgrade and now have 0.13.0 Restarted vscode and annotations appeared.
Question: I'm trying to use scipy's quadratic_assignment function but I can't understand how the output can describe an optimal solution. Here is a minimal example where I compare a small matrix to itself: import numpy as np from scipy.optimize import quadratic_assignment # Parameters n = 5 p = np.log(n)/n # Approx. 32% # Definitions A = np.random.rand(n,n)<p # Quadratic assignment res = quadratic_assignment(A, A) print(res.col_ind)  and the results seem to be random assignments: [3 0 1 4 2] [3 2 4 1 0] [3 2 1 0 4] [4 3 1 0 2] [2 3 0 1 4] ...  However, according to the docs col_ind is supposed to be the Column indices corresponding to the best permutation found of the nodes of B. Since the input matrices are equal (B==A), I would thus expect the identity assignment [0 1 2 3 4] to pop out. Changing n to larger values does not help. Is there something I am getting wrong?

Answer: The quadratic_assignment function (approximately) solves two types of problem: quadratic assignment and graph matching. These are mathematically very similar: the difference is that quadratic assignment involves minimising an objective function, whereas graph matching involves maximising that same function. (Reference: scipy docs). To distinguish between these two problems, the function accepts an option maximize (passed as a key:value pair in the options argument of the function). If not supplied, this defaults to False (i.e. the solution corresponds to the minimum of the function, i.e. the quadratic assignment problem rather than the graph matching problem). quadratic_assignment(A, B) # solves quadratic assignment quadratic_assignment(A, B, options={'maximize': False}) # solves quadratic assignment quadratic_assignment(A, B, options={'maximize': True}) # solves graph matching  Now, when you say "Since the input matrices are equal (B==A), I would thus expect the identity assignment [0 1 2 3 4] to pop out" – that's what would be expected for graph matching, i.e. the identity assignment is indeed the permutation of B that matches A better than any other (or strictly speaking: at least as well as any other). So you would get your expected result by specifying options={'maximize': True} so that you get the solution to the graph matching problem rather than the default quadratic assignment: import numpy as np from scipy.optimize import quadratic_assignment # Parameters n = 5 p = np.log(n)/n # Approx. 32% # Definitions A = np.random.rand(n,n)<p # Graph matching res = quadratic_assignment(A, A, options={'maximize': True}) print(res.col_ind)  When I run this, I get the expected [0 1 2 3 4] around 97% of the time. It's not 100% because (a) the algorithm is approximate and "not guaranteed to be optimal" (per the docs) and (b) the way that you construct A could create degenerate cases where there are multiple equally good solutions.
Question: I want to find the dark gray diagonal line in this  the background is quite noisy and has a gradient in brightness. (The line is barely visible when opening the .png but if I read it as grayscale the line becomes more pronounced.) I tried different combinations of bluring, thresholding and canny edge detection. The best I could come up with was: img_blur = cv.bilateralFilter(img, 3, 120, 120) thresh = cv.adaptiveThreshold(img_blur, 255, cv.ADAPTIVE_THRESH_GAUSSIAN_C, cv.THRESH_BINARY, 501, 3)  which results in this . Here there is still quite alot of noise in the background and the line is interrupted. I tried some morphological operations (dilate, erode, open, close) but with no real improvement. Applying something like lines = cv.HoughLines(thresh, rho=1, theta=np.pi / 180, threshold=130) left me with this . Which is what I wanted, but the threshold of 130 doesn't work for similar images and will find no or too many lines.

Answer: You can try to first detect ridges in the image using a Hessian, and then threshold. This seems to work okay on your example image. import cv2 import numpy as np import matplotlib.pyplot as plt from skimage.feature import hessian_matrix, hessian_matrix_eigvals def detect_ridges(img: np.ndarray, sigma: int = 3) -> np.ndarray: img = cv2.equalizeHist(img.astype(np.uint8)) elements = hessian_matrix(img, sigma, use_gaussian_derivatives=False) eigvals = hessian_matrix_eigvals(elements) cv2.normalize(eigvals[0], eigvals[0], 0, 255, cv2.NORM_MINMAX).astype(np.uint8) return eigvals[0] original_image = cv2.imread("zkN7m.png", cv2.IMREAD_GRAYSCALE) ridges = detect_ridges(original_image) thresholded = cv2.adaptiveThreshold((255-ridges).astype(np.uint8), 255, cv2.ADAPTIVE_THRESH_GAUSSIAN_C, cv2.THRESH_BINARY, 3, 2) plt.imshow(thresholded, cmap="bone")
Question: In pandas 2 with the following code: for time in tfo_dates: dt=pd.to_datetime(time) indx_time.append(df.index.get_loc(dt,method='nearest'))  I get this error: TypeError: get_loc() got an unexpected keyword argument 'method'  This worked in version 1.5 but if we look at the version 2 documentation there is no method argument anymore. What method I can use now to get nearest index of timestamp inside time index list?

Answer: IIUC, no need for a loop : matches = df.index.get_indexer(pd.to_datetime(tfo_dates), method="nearest") out = df.iloc[matches]  Output : print(out) B A 2023-09-01 17:05:30 0.548814 2023-09-03 13:05:30 0.461479 2023-09-04 23:05:30 0.681820  Used inputs :  B A 2023-09-01 17:05:30 0.548814 2023-09-01 19:05:30 0.715189 2023-09-01 21:05:30 0.602763 2023-09-01 23:05:30 0.544883 2023-09-02 01:05:30 0.423655 ... ... 2023-09-04 15:05:30 0.617635 2023-09-04 17:05:30 0.612096 2023-09-04 19:05:30 0.616934 2023-09-04 21:05:30 0.943748 2023-09-04 23:05:30 0.681820 [40 rows x 1 columns]   np.random.seed(0) dtr = pd.date_range("20230901 170530", "20230905 002000", freq="2H") df = pd.DataFrame({"B": np.random.rand(len(dtr))}, index=pd.Index(dtr, name="A")) tfo_dates = ["2023-09-01 12:00:00", "2023-09-03 13:00:00", "2023-09-05 14:00:00"]
Question: I have two arrays of different sizes and want to determine where elements of one array can be found in the other array. I would like to be able to allow for a tolerance between elements. The goal would be something like this: array1 = [1, 2, 3, 4, 5, 6] array2 = [2, 8, 1.00001, 1.1] places = [mystery function](array1, array2, tolerance = 0.001)  returning indices places = [0,1] in array1. The closest I can get is with np.isin, which allows for arrays of different sizes and orders, but not the tolerance. (have also tried np.allclose but the shape and order mismatch is an issue there). Of course this can be done with a loop, but my actual arrays are thousands of elements long so a loop is not practical. (it also doesn't have to be a numpy function--really just anything more efficient than a loop) Thanks in advance for the help!

Answer: With custom function to find the absolute difference between 2 arrays and positions which fit the given tolerance: array1 = np.array([1, 2, 3, 4, 5, 6]) array2 = np.array([2, 8, 1.00001, 1.1]) tolerance = 0.001 def argwhere_close(a, b, tol): return np.where(np.any(np.abs(a - b[:, None]) <= tolerance, axis=0))[0] print(argwhere_close(array1, array2, tolerance))   [0 1]
Question: I am working on a research project where I need to summarize news articles using the Google Palm2 Text Generation Model. I have encountered an issue with certain news articles in my dataset where I'm getting empty responses along with safety attributes that block the output. Here is the code I'm using: from vertexai.language_models import TextGenerationModel parameters = { # default values 'max_output_tokens': 256, 'temperature': 0.0, 'top_p': 1.0, 'top_k': 40, } prompt = "..." model = TextGenerationModel.from_pretrained('text-bison@001') response = model.predict( prompt, **parameters, )  The following is an example prediction: Prediction(predictions=[{'content': '', 'citationMetadata': None, 'safetyAttributes': {'blocked': True, 'errors': [253.0]}}], deployed_model_id='', model_version_id='', model_resource_name='', explanations=None)  The issue seems to be related to safety parameters preventing the model from generating a summary for certain news articles. I've been trying to find documentation on how to configure these safety parameters using the Python API, but I could not locate the relevant information. Could someone please provide guidance on how to set the safety parameters for the TextGenerationModel? Any help or pointers to documentation would be greatly appreciated. Thank you!

Answer: I'm not sure about Vertex AI but you can set the safety_settings of the PaLM model (from google generative AI) by the following: import google.generativeai as palm completion = palm.generate_text( model=model, prompt=prompt, safety_settings=[ { "category": safety_types.HarmCategory.HARM_CATEGORY_DEROGATORY, "threshold": safety_types.HarmBlockThreshold.BLOCK_NONE, }, { "category": safety_types.HarmCategory.HARM_CATEGORY_VIOLENCE, "threshold": safety_types.HarmBlockThreshold.BLOCK_NONE, }, ] )  You should checkout this guide to get complete details of the safety catalogue and how to set threshold for each category as there are multiple categories and different threshold levels. NOTE: To use the PaLM API from generative AI, you'd need to install it first via: pip install -q google-generativeai  and then set an API key which you'll get from here: import google.generativeai as palm palm.configure(api_key='YOUR_API_KEY')  and then to access the same text-bison-001 model: models = [m for m in palm.list_models() if 'generateText' in m.supported_generation_methods] model = models[0].name # use this model on the first code snippet print(model) # prints 'models/text-bison-001'
Question: recently I've made an automated file sorter based on file extension using Python and Tkinter GUI module, after I was done I compiled the the Python code into an executable using PyInstaller via windows terminal, then I put a "README" text file into the executable folder and compiled that folder as a setup executable using Inno Setup Compiler, it seems to work fine on my computer but when I try to send it to someone, the Anti-virus of the user detects it as a malware and blocks it or sets it in "Quarantine". So I'd really appreciate it if I can resolve this problem so other users can benefit from it without any problems. Thank you for your time <3 • Note: I am using Python 3.8 and customtkinter along with the normal Tkinter (both latest versions) I didn't know how to resolve it and couldn't think of a better place to ask than here.

Answer: This is both an interesting question and a question without answer ;-) You indeed have a real problem and correctly explain what happens. Unfortunately converting a Python program to a Windows executable will almost always raises anti-malware warnings. The underlying reason, is that your executable uses a bootstrap code, that has to prepare an environment for the embedded executable to execute the Python script. And most anti-malware tools choke when they see a program extract something from its data to execute it, because it is a well known malware pattern where a genuine program is wrapped into something that first tries to spread a virus and in the end execute the original code. That means that I cannot propose any way to really solve your problem :-( What can be done:  if you only give your program to friends, just explain them that it will be falsely detected - you even now can explain the reason - and that they have to set an exception in their anti-virus if you want to distibute it wildly, and work in a major organization, contact the major anti-virus companies to explain how the program was built (they may require the source and the build instructions) so that they explicitely allow it if neither of the previous ways are applicable, just explain the problem and provide the source code and build instructions to allow power users to build from source  I am aware that no of those workarounds is a nice solution, but it is the best I can give you...
Question: I have a situation like following:     ID Old value.Carbon Old value.Dioxide New value.Carbon New Value.Dioxide     123 34.89 13.45 56.66 11.11   456 12.13 55.66 66.88 12.33     My output that i want should be:( i want the column to be transposed to rows like following, i have renamed the fields as Old Value. and New Value. because i thought this might help but if you have any other ideas i am more than happy to try something different. Thanks a lot for all the help!     ID Field New Value Old Value     123 Carbon 56.66 34.89   123 Dioxide 11.11 13.45   456 Dioxide 12.33 12.33   456 Carbon 66.88 12.13    

Answer: Using stack and a MultiIndex to reshape: tmp = df.set_index('ID') out = (tmp.set_axis(tmp.columns.str.capitalize() .str.split('\.', expand=True), axis=1) .rename_axis(columns=(None, 'Field')) .stack() .reset_index() )  Output:  ID Field Old value New value 0 123 carbon 34.89 56.66 1 123 dioxide 13.45 11.11 2 456 carbon 12.13 66.88 3 456 dioxide 55.66 12.33
Question: I came across this documentation in the ast module for a version of the try/except block with an extra asterisk. The documentation doesn't explain what it is, and gives a completely generic example:  class ast.TryStar(body, handlers, orelse, finalbody) try blocks which are followed by except* clauses. The attributes are the same as for Try but the ExceptHandler nodes in handlers are interpreted as except* blocks rather then except. print(ast.dump(ast.parse(""" try: ... except* Exception: ... """), indent=4))   What is except* and what is it for? Is it deprecated, or up-and-coming? (And perhaps more importantly, what is the feature called? except-star? except-glob? except-asterisk? try-star?)

Answer: except*(aka except-star) is one of the feature/syntax introduced in Python 3.11 and it is used to handle ExceptionGroup. Quoting from the official documentation  except* clause The except* clause(s) are used for handling ExceptionGroups. The exception type for matching is interpreted as in the case of except, but in the case of exception groups we can have partial matches when the type matches some of the exceptions in the group. This means that multiple except* clauses can execute, each handling part of the exception group. Each clause executes at most once and handles an exception group of all matching exceptions. Each exception in the group is handled by at most one except* clause, the first that matches it. try: raise ExceptionGroup("eg", [ValueError(1), TypeError(2), OSError(3), OSError(4)]) except* TypeError as e: print(f'caught {type(e)} with nested {e.exceptions}') except* OSError as e: print(f'caught {type(e)} with nested {e.exceptions}') # caught <class 'ExceptionGroup'> with nested (TypeError(2),) # caught <class 'ExceptionGroup'> with nested (OSError(3), OSError(4)) # + Exception Group Traceback (most recent call last): # | File "<stdin>", line 2, in <module> # | ExceptionGroup: eg # +-+---------------- 1 ---------------- # | ValueError: 1 # +------------------------------------
Question: Consider the following array: arr = np.array( [ [10, np.nan], [20, np.nan], [np.nan, 50], [15, 20], [np.nan, 30], [np.nan, np.nan], [10, np.nan], ] )  For every cell in each column in arr I need to find the distance to the next non-NaN value. That is, the expected outcome should look like this: expected = np.array( [ [1, 2], [2, 1], [1, 1], [3, 1], [2, np.nan], [1, np.nan], [np.nan, np.nan] ] ) 

Answer: Using pandas, you can compute a reverse cumcount, with mask and shift: out = (pd.DataFrame(arr).notna()[::-1] .apply(lambda s: s.groupby(s.cumsum()).cumcount().add(1) .where(s.cummax()).shift()[::-1]) .to_numpy() )  Output: array([[ 1., 2.], [ 2., 1.], [ 1., 1.], [ 3., 1.], [ 2., nan], [ 1., nan], [nan, nan]])
Question: I have a data frame with 3 index levels:  d a b c 1 9 4 1 2 8 2 4 3 7 5 2 4 6 4 5 5 5 6 3 6 4 5 6 7 3 7 4 8 2 6 7 9 1 8 5  and I have a multi index object with only 2 levels: MultiIndex([(1, 9), (2, 8), (3, 7), (4, 6), (9, 1)], names=['a', 'b'])  How can I select the entries on the data frame that match this multi index? Toy code: import pandas df1 = pandas.DataFrame( dict( a = [1,2,3,4,5,6,7,8,9], b = [9,8,7,6,5,4,3,2,1], c = [4,2,5,4,6,5,7,6,8], d = [1,4,2,5,3,6,4,7,5], ) ).set_index(['a','b','c']) select_this = multi_idx = pandas.MultiIndex.from_tuples([(1, 9), (2, 8), (3, 7), (4, 6), (9, 1)], names=['a', 'b']) selected = df1.loc[select_this] print(select_this) print(df1) print(selected)  which produces ValueError: operands could not be broadcast together with shapes (5,2) (3,) (5,2). What I want to do can be achieved with selected = df1.reset_index('c').loc[select_this].set_index('c', append=True)  However, this forces me to do this extra reset_index and then set_index. I want to avoid this.

Answer: You can filter different levels by Index.difference and filter by boolean indexing with Index.isin: lvl = df1.index.names.difference(multi_idx.names) out = df1[df1.index.droplevel(lvl).isin(multi_idx)] print (out) d a b c 1 9 4 1 2 8 2 4 3 7 5 2 4 6 4 5 9 1 8 5