eshangj/stackoverflow_q_and_a_sample huggingface dataset:
Question: Basic task of making code that can find the sum & average of 10 numbers from the user. My current situation so far: Sum = 0 print("Please Enter 10 Numbers\n") for i in range (1,11): num = int(input("Number %d =" %i)) sum = Sum + num avg = Sum / 10  However, I want to make it so that if the user inputs an answer such as "Number 2 = 1sd" it doesn't stop the code immediately. Instead I'm looking for it to respond as "Invalid input. Try again" and then it restarts from the failed input. e.g. "Number 2 = 1sd" "Invalid input. Try again." "Number 2 = ..."  How would I achieve that?

Answer: You can output an error message and prompt for re-entry of the ith number by prompting in a loop which outputs an error message and reprompts on user input error. This can be done by handling the ValueError exception that is raised when the input to the int() function is an invalid integer object string initializer such as as "1sd". That can be done by catching that exception, outputting Invalid input. Try again. and continuing when that occurs. One possible resulting code that may satisfy your requirements is: Sum = 0 print("Please Enter 10 Numbers\n") for i in range (1,11): num = None while num is None: try: num = int(input("Number %d =" %i)) except ValueError: print('Invalid input. Try again.') sum = Sum + num avg = Sum / 10
Question: How do I read sys.stdin, but ignoring decoding errors? I know that sys.stdin.buffer exists, and I can read the binary data and then decode it with .decode('utf8', errors='ignore'), but I want to read sys.stdin line by line. Maybe I can somehow reopen the sys.stdin file but with errors='ignore' option?

Answer: Found three solutions from here as Mark Setchell mentioned. import sys import io def first(): with open(sys.stdin.fileno(), 'r', errors='ignore') as f: return f.read() def second(): sys.stdin = io.TextIOWrapper(sys.stdin.buffer, errors='ignore') return sys.stdin.read() def third(): sys.stdin.reconfigure(errors='ignore') return sys.stdin.read() print(first()) #print(second()) #print(third())  Usage: $ echo 'a\x80b' | python solution.py ab
Question: I've just installed Anaconda in my new laptop, and created an environment with geopandas installed in it. I've tried to upload the world map that comes with geopandas through the following code: import geopandas as gpd world = gpd.read_file(gpd.datasets.get_path('naturalearth_lowres'))  But I obtain the following error message: File ~/anaconda3/envs/mapas_test/lib/python3.8/site-packages/shapely/geometry/base.py:854, in BaseMultipartGeometry.__array_interface__(self) 851 @property 852 def __array_interface__(self): 853 """Provide the Numpy array protocol.""" --> 854 raise NotImplementedError("Multi-part geometries do not themselves " 855 "provide the array interface") NotImplementedError: Multi-part geometries do not themselves provide the array interface  Since this error has never appeared in my old laptop, I guess it is related to some problem during installation, but I could be wrong. Here are the technical details about the installation. OS: Ubuntu 22.04.1 Python version: 3.9.12 Conda version 4.13.0 geopandas version 0.9.0 Shapely version 1.7.1 And not sure if it is relevant, but the only other package installed in the environment is jupyter version 1.0.0

Answer: This is caused by incompatibility of shapely 1.7 and numpy 1.23. Either update shapely to 1.8 or downgrade numpy, otherwise it won't work.
Question: I'm new to PySpark and I see there are two ways to select columns in PySpark, either with ".select()" or ".withColumn()". From what I've heard ".withColumn()" is worse for performance but otherwise than that I'm confused as to why there are two ways to do the same thing. So when am I supposed to use ".select()" instead of ".withColumn()"? I've googled this question but I haven't found a clear explanation.

Answer: Using: df.withColumn('new', func('old'))  where func is your spark processing code, is equivalent to: df.select('*', func('old').alias('new')) # '*' selects all existing columns  As you see, withColumn() is very convenient to use (probably why it is available), however as you noted, there are performance implications. See this post for details: Spark DAG differs with 'withColumn' vs 'select'
Question: When running the code below: import lightgbm as lgb params = {'num_leaves': 38, 'min_data_in_leaf': 50, 'objective': 'regression', 'max_depth': -1, 'learning_rate': 0.1, 'device': 'gpu' } trn_data = lgb.Dataset(x_train, y_train) val_data = lgb.Dataset(x_test, y_test) model = lgb.train(params, trn_data, 20000, valid_sets=[trn_data, val_data], verbose_eval=300, early_stopping_rounds=1000)  I get the follow errors: train() got an unexpected keyword argument 'verbose_eval' train() got an unexpected keyword argument 'early_stopping_rounds'  It is important to note that I run this on GPU. When running it on CPU i do not get this error. Has anyone got an idea how I can incorporate a verbose output and early stopping rounds when running Lightgbm on GPU??

Answer: For Lightgbm on GPU you can check the official documentation. On documentation there are no configuration option as vebose_ecal and early_stopping_rounds Official Documentation Also you can check this link Running LightGBM on GPU with python
Question: I had a performance issue in a numpy project and then I realized that about 3 fourth of the execution time is wasted on a single line of code: error = abs(detected_matrix[i, step] - original_matrix[j, new]) and when I have changed the line to error = abs(original_matrix[j, new] - detected_matrix[i, step]) the problem has disappeared. I relized that the type of original_matrix was float64 and type of detected_matrix was float32. By changing types of either of these two varibles the problem solved. I was wondering that if this is a well known issue? Here is a sample code that represents the problem from timeit import timeit import numpy as np f64 = np.array([1.0], dtype='float64')[0] f32 = np.array([1.0], dtype='float32')[0] timeit_result = timeit(stmt="abs(f32 - f64)", number=1000000, globals=globals()) print(timeit_result) timeit_result = timeit(stmt="abs(f64 - f32)", number=1000000, globals=globals()) print(timeit_result)  Output in my computer: 2.8707289 0.15719420000000017  which is quite strange.

Answer: TL;DR: Please use Numpy >= 1.23.0. This problem has been fixed in Numpy 1.23.0 (more specifically the version 1.23.0-rc1). This pull request rewrites the scalar math logic so to make it faster in many cases including in your specific use-case. With version 1.22.4, the former code is 10 times slower than the second one. This is also true for earlier versions like the 1.21.5. In the 1.23.0, the former is only 10%-15% slower but both takes a very small time: 140 ns/operation versus 122 ns/operation. The small difference is due to a slightly different path taken in the type-checking part of the code. For more information about this low-level behavior, please read this post. Note that iterating over Numpy items it not meant to be very fast, nor operating on Numpy scalar. If your code is limited by that, please consider converting Numpy scalar into Python ones as stated in the 1.23.0 release notes:  Many operations on NumPy scalars are now significantly faster, although rare operations (e.g. with 0-D arrays rather than scalars) may be slower in some cases. However, even with these improvements users who want the best performance for their scalars, may want to convert a known NumPy scalar into a Python one using scalar.item().  An even faster solution is to use Numba/Cython in this case or just to try to vectorize the encompassing loop if possible.
Question: I have a proto file defined as: syntax = "proto3"; import "google/protobuf/struct.proto"; package generic.name; message Message { uint32 increment = 1; google.protobuf.Value payload = 2; }  I have figured out how to make this work if I swap the payload type from Value for Struct:  struct = Struct() struct.update({"a": 1}) msg = Message(payload=struct, increment=1)  However I cannot work out how to use the Value type in python. The python documentation for the protobuf Value field seems lacking compared to the other languages. Ultimately, all I want is to be able to have the payload data structure able to take a few different types (strings, ints, none, dict). What is the best way of achieving this?

Answer: Here's an example: from foo_pb2 import Bar from google.protobuf.struct_pb2 import Struct bar = Bar() bar.payload.string_value="Freddie" print(bar) print(bar.SerializeToString()) bar.payload.bool_value=True print(bar) print(bar.SerializeToString()) # No need to initialize the Struct bar.payload.struct_value.update({"foo":"bar"}) bar.payload.struct_value["baz"]=5 print(bar) print(bar.SerializeToString())
Question: I'm running an Azure function locally, from VSCode, that outputs a string to a blob. I'm using Azurite to emulate the output blob container. My function looks like this: import azure.functions as func def main(mytimer: func.TimerRequest, outputblob:func.Out[str]): outputblob.set("hello")  My function.json: { "scriptFile": "__init__.py", "bindings": [ { "name": "mytimer", "type": "timerTrigger", "direction": "in", "schedule": "0 * * * * *" }, { "name": "outputblob", "type": "blob", "dataType": "string", "direction": "out", "path": "testblob/hello" } ] }  In local.settings.json, I've set "AzureWebJobsStorage": "UseDevelopmentStorage=true". The problem is, when I run the function and check in Azure Storage Explorer, the container is created (testblob) (along with 2 other containers: azure-webjobs-hosts and azure-webjobs-secrets) but it is empty and Azure Storage Explorer displays an error message when I refresh : The first argument must be of type string or an instance of Buffer, ArrayBuffer, or Array or an Array-like Object.Received undefined The function runs and doesn't return any error message. When I use a queue instead of a blob as output, it works and I can see the string in the emulated queue storage. When I use the blob storage in my Azure subscription instead of the emulated blob, it works as well, a new blob is created with the string. I've tried the following:  clean and restart Azurite several times replace "UseDevelopmentStorage=true" by the connection string of the emulated storage reinstall Azure Storage Explorer  I keep getting the same error message. I'm using Azure Storage Explorer Version 1.25.0 on Windows 11. Thanks for any help!

Answer: It looks like this is a known issue with the latest release (v1.25.0) of Azure Storage Explorer version see: https://github.com/microsoft/AzureStorageExplorer/issues/6008 Simplest solution is to uninstall and re-install an earlier version: https://github.com/microsoft/AzureStorageExplorer/releases/tag/v1.24.3
Question: I couldn't find a definitive answer for this seemingly simple question. If I write a method like this: public Integer getAnInt() { int[] i = {4}; return i[0]; }  is the return value autoboxed into an Integer, or does it depend on what happens to the value after it's returned (e.g. whether the variable it is assigned to is declared as an Integer or int)?

Answer: Yes, boxed It will be (auto)boxed in the bytecode (.class file) because it's part of the public API, so other code might depend on the return value being an Integer. The boxing and unboxing might be removed at runtime by the JITter under the right circumstances, but I don't know if it does that sort of thing.
Question: Similar to this question, I'd like to edit/set the description of a run via code, instead of editing it via UI. To clarify, I don't want to set the description of my entire experiment, only of a single run. 

Answer: There are two ways to set the description. 1. description parameter You can set a description using a markdown string for your run in mlflow.start_run() using description parameter. Here is an example. if __name__ == "__main__": # load dataset and other stuff run_description = """ ### Header This is a test **Bold**, *italic*, ~~strikethrough~~ text. [And this is an example hayperlink](http://example.com/). """ with mlflow.start_run(description=run_description) as run: # train model and other stuff  2. mlflow.note.content tag You can set/edit run names by setting the tag with the key mlflow.note.content, which is what the UI (currently) does under the hood. if __name__ == "__main__": # load dataset and other stuff run_description = """ ### Header This is a test **Bold**, *italic*, ~~strikethrough~~ text. [And this is an example hayperlink](http://example.com/). """ tags = { 'mlflow.note.content': run_description } with mlflow.start_run(tags=tags) as run: # train model and other stuff  Result   If you set description parameter and mlflow.note.content tag in mlflow.start_run(), you'll get this error. Description is already set via the tag mlflow.note.content in tags. Remove the key mlflow.note.content from the tags or omit the description.
Question: I am using regex over a whole document to catch the spans in which such regex occurs: import spacy import re nlp = spacy.load("en_core_web_sm") doc = nlp("The United States of America (USA) are commonly known as the United States (U.S. or US) or America.") expression = r"[Uu](nited|\.?) ?[Ss](tates|\.?)" for match in re.finditer(expression, doc.text): start, end = match.span() span = doc.char_span(start, end) # This is a Span object or None # if match doesn't map to valid token sequence if span is not None: print("Found match:", span.text)  There is a way to get the span (list of tokens) corresponding to the regex match on the doc even if the boundaries of the regex match do not correspond to token boundaries. See: How can I expand the match to a valid token sequence? In https://spacy.io/usage/rule-based-matching So far so good. Now that I have a collectuon of spans how do I convert them into entities? I am aware of the entity ruler: The EntityRuler is a pipeline component (see also the link above) but that entityruler takes patterns as inputs to search in the doc and not spans. If I want to use regex over the whole document to get the collection os spans I want to convert into ents what is the next step here? Entityruler? How? Or something else? Put simpler: nlp = spacy.load("en_core_web_sm") doc = nlp("The aplicable law is article 102 section b sentence 6 that deals with robery")  I would like to generate an spacy ent (entity) out of doc[5,10] with label "law" in order to be able to: A) loop over all the law entities in the texts B) use the visualizer to display the different entities contained in the doc

Answer: The most flexible way to add spans as entities to a doc is to use Doc.set_ents: from spacy.tokens import Span span = doc.char_span(start, end, label="ENT") doc.set_ents(entities=[span], default="unmodified")  Use the default option to specify how to set all the other tokens in the doc. By default the other tokens are set to O, but you can use default="unmodified" to leave them untouched, e.g. if you're adding entities incrementally. https://spacy.io/api/doc#set_ents
Question: I have some points in legacy code (python library: music21) that uses a lot of overloading and Generic variables to show/typecheck that all subelements in a t.Sequence belong to a particular type. There are a lot of @overload decorators to show how different attributes can return different values. At this point the functions work as they should, but a number of PRs in the past have broken the introspection that other devs require. The code is extensively tested, but the inferred types by checkers such as mypy and PyCharm are not. Is there a way to run testing on inferred types? Something like: SomeClassType = typing.TypeVar('SomeClassType', bound='SomeClass') class SomeClass: pass class DerivedClass(SomeClass): pass class MyIter(typing.Sequence[typing.Type[SomeClassType]]): def __init__(self, classType: typing.Type[SomeClassType]): self.classType = classType # ------- type_checks.py... derived_iterator = MyIter(DerivedClass) # this is the line that I don't know exists... typing_utilities.assert_reveal_type_eq(derived_iterator, MyIter[DerivedClass]) # or as a string 'MyIter[DerivedClass]'  mypy's reveal_type seems like it would be helpful here, but I can't seem to find any integration into a testing system, etc. Thanks!

Answer: The function you are looking for actually exists. But it is called differently: First, define a type test: from typing_extensions import assert_type def function_to_test() -> int: pass # this is a positive test: we want the return type to be int assert_type(function_to_test(), int) # this is a negative test: we don't want the return type to be str assert_type(function_to_test(), str) # type: ignore  Then run mypy on the file: mypy --strict --warn-unused-ignores. The positive tests that fail are simply reported as a mypy error and the negative tests that fail are reported as 'Unused "type: Ignore" comment'. The package typing_extensions is installed alongside mypy. Source: https://typing.readthedocs.io/en/latest/source/quality.html
Question: Basically, the expression 0.4 * a is consistently, and surprisingly, significantly faster than a * 0.4. a being an integer. And I have no idea why. I speculated that it is a case of a LOAD_CONST LOAD_FAST bytecode pair being "more specialized" than the LOAD_FAST LOAD_CONST and I would be entirely satisfied with this explanation, except that this quirk seems to apply only to multiplications where types of multiplied variables differ. (By the way, I can no longer find the link to this "bytecode instruction pair popularity ranking" I once found on github, does anyone have a link?) Anyway, here are the micro benchmarks: $ python3.10 -m pyperf timeit -s"a = 9" "a * 0.4" Mean +- std dev: 34.2 ns +- 0.2 ns  $ python3.10 -m pyperf timeit -s"a = 9" "0.4 * a" Mean +- std dev: 30.8 ns +- 0.1 ns  $ python3.10 -m pyperf timeit -s"a = 0.4" "a * 9" Mean +- std dev: 30.3 ns +- 0.3 ns  $ python3.10 -m pyperf timeit -s"a = 0.4" "9 * a" Mean +- std dev: 33.6 ns +- 0.3 ns  As you can see - in the runs where the float comes first (2nd and 3rd) - it is faster. So my question is where does this behavior come from? I'm 90% sure that it is an implementation detail of CPython, but I'm not that familiar with low level instructions to state that for sure.

Answer: It's CPython's implementation of the BINARY_MULTIPLY opcode. It has no idea what the types are at compile-time, so everything has to be figured out at run-time. Regardless of what a and b may be, BINARY_MULTIPLY ends up inoking a.__mul__(b). When a is of int type int.__mul__(a, b) has no idea what to do unless b is also of int type. It returns Py_RETURN_NOTIMPLEMENTED (an internal C constant). This is in longobject.c's CHECK_BINOP macro. The interpreter sess that, and effectively says "OK, a.__mul__ has no idea what to do, so let's give b.__rmul__ a shot at it". None of that is free - it all takes time. float.__mul__(b, a) (same as float.__rmul__) does know what to do with an int (converts it to float first), so that succeeds. But when a is of float type to begin with, we go to float.__mul__ first, and that's the end of it. No time burned figuring out that the int type doesn't know what to do. The actual code is quite a bit more involved than the above pretends, but that's the gist of it.
Question: I've two PyTorch tensors mask = torch.ones(1024, 64, dtype=torch.float32) indices = torch.randint(0, 64, (1024, ))  For every ith row in mask, I want to set all the elements after the index specified by ith element of indices to zero. For example, if the first element of indices is 50, then I want to set mask[0, 50:]=0. Is it possible to achieve this without using for loop? Solution with for loop: for i in range(mask.shape[0]): mask[i, indices[i]:] = 0 

Answer: You can first generate a tensor of size (1024x64) where each row has numbers arranged from 0 to 63. Then apply a logical operation using the indices reshaped as (1024x1) mask = torch.ones(1024, 64, dtype=torch.float32) indices = torch.randint(0, 64, (1024, 1)) # Note the dimensions mask[torch.arange(0, 64, dtype=torch.float32).repeat(1024,1) >= indices] = 0
Question: Minimal example: pip install tensorflow==2.9.1 non-existing==1.2.3  Defaulting to user installation because normal site-packages is not writeable Looking in indexes: https://pypi.org/simple, https://pypi.ngc.nvidia.com Collecting tensorflow==2.9.1 Downloading tensorflow-2.9.1-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (511.7 MB) ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 511.7/511.7 MB 7.6 MB/s eta 0:00:00 ERROR: Could not find a version that satisfies the requirement non-existing==1.2.3 (from versions: none) ERROR: No matching distribution found for non-existing==1.2.3  So pip downloads the (rather huge) TensorFlow first, only to then tell me that non-existing does not exist. Is there a way to make it fail earlier, i.e., print the error and quit before downloading?

Answer: I'm afraid there's no straightforward way of handling it. I ended up writing a simple bash script where I check the availability of packages using pip's index command: check_packages_availability () { while IFS= read -r line || [ -n "$line" ]; do package_name="${line%%=*}" package_version="${line#*==}" if ! pip index versions $package_name | grep "$package_version"; then echo "package $line not found" exit -1 fi done < requirements.txt } if ! check_packages_availability; then pip install -r requirements.txt fi  This is a hacky solution but may work. For every package in requirements.txt this script tries to retrieve information about it and match the specified version. If everything's alright it starts installing them.  Or you can use poetry, it handles resolving dependencies for you, for example: pyproject.toml [tool.poetry] name = "test_missing_packages" version = "0.1.0" description = "" authors = ["funnydman"] [tool.poetry.dependencies] python = "^3.10" tensorflow = "2.9.1" non-existing = "1.2.3" [build-system] requires = ["poetry-core>=1.0.0"] build-backend = "poetry.core.masonry.api"  At the resolving stage it throws exception without installing/downloading packages: Updating dependencies Resolving dependencies... (0.2s) SolverProblemError Because test-missing-packages depends on non-existing (1.2.3) which doesn't match any versions, version solving failed.
Question: I am experiencing an obscure (to me) effect of dundered scoping, and trying to work out the rule for it: #!/usr/bin/env python3 stuff = "the things" __MORE_STUFF = "even more" class Thing: def __init__(self): global __MORE_STUFF # doesn't help print(stuff) # is OK print(__MORE_STUFF) # fail! Thing()  results in $ python3 dunder.py the things Traceback (most recent call last): File "dunder.py", line 12, in <module> Thing() File "dunder.py", line 10, in __init__ print(__MORE_STUFF) # fail! NameError: name '_Thing__MORE_STUFF' is not defined  What is supposed to be a module-global variable is being treated as a class-level property - which, being undefined, is flagged as being undefined. I've been trying to look through the documentation for this, but I cannot seem to work out what the rule is. Can anyone point me to the appropriate documentation?

Answer: The documentation refers to such names as class-private names:  __* Class-private names. Names in this category, when used within the context of a class definition, are re-written to use a mangled form to help avoid name clashes between “private” attributes of base and derived classes. See section Identifiers (Names).  Since subclassing isn't something that applies to modules (at least, not to the extent that the language provides any tools or recommendations regarding how you might go about it), there's no need to use class-private names where _-prefixed names will do. #!/usr/bin/env python3 stuff = "the things" _MORE_STUFF = "even more" class Thing: def __init__(self): print(stuff) # is OK print(_MORE_STUFF) Thing()