eshangj/stackoverflow_q_and_a_sample huggingface dataset:
Question: I have a deeply nested tuple of nested lists as follows: ip = (array([[[ 50, 73]], [[ 50, 107]], [[ 55, 108]], [[ 55, 121]], [[978, 87]], [[977, 86]], [[977, 73]]], dtype=int32), array([[[ 669, 3]], [[ 668, 4]], [[ 667, 4]], [[1033, 71]], [[1035, 69]], [[1035, 4]], [[ 848, 4]], [[ 847, 3]], [[ 813, 3]], [[ 718, 4]], [[ 717, 3]]], dtype=int32), array([[[ 17, 3]], [[ 16, 4]], [[ 0, 4]], [[ 0, 49]], [[197, 49]], [[197, 8]], [[ 84, 4]], [[ 83, 3]]], dtype=int32))  The length of the main tuple in above example is 3. I want to perform a 2 level sorting on the above structure. First I want to sort all the 3 elements in the main list in increasing order based on the first value of nested list. So in the above case the third element will come first as it has the lowest value of the first element i.e. 0. Second should be the first element as it has the second lowest value of 50and last should be the third element as it has the third lowest value of 1035. The output of the first level sorting should be: op = (array([[[ 17, 3]], [[ 16, 4]], [[ 0, 4]], [[ 0, 49]], [[197, 49]], [[197, 8]], [[ 84, 4]], [[ 83, 3]]], dtype=int32), array([[[ 50, 73]], [[ 50, 107]], [[ 55, 108]], [[ 55, 121]], [[978, 87]], [[977, 86]], [[977, 73]]], dtype=int32), array([[[ 669, 3]], [[ 668, 4]], [[ 667, 4]], [[1033, 71]], [[1035, 69]], [[1035, 4]], [[ 848, 4]], [[ 847, 3]], [[ 813, 3]], [[ 718, 4]], [[ 717, 3]]], dtype=int32), )  Now I want to perform the same sorting again on the above op but instead of the first value of the nested list I want to sort based on the second value of the nested list. So now the final output would be as follows: final_op = (array([[[ 17, 3]], [[ 16, 4]], [[ 0, 4]], [[ 0, 49]], [[197, 49]], [[197, 8]], [[ 84, 4]], [[ 83, 3]]], dtype=int32), array([[[ 669, 3]], [[ 668, 4]], [[ 667, 4]], [[1033, 71]], [[1035, 69]], [[1035, 4]], [[ 848, 4]], [[ 847, 3]], [[ 813, 3]], [[ 718, 4]], [[ 717, 3]]], dtype=int32), array([[[ 50, 73]], [[ 50, 107]], [[ 55, 108]], [[ 55, 121]], [[978, 87]], [[977, 86]], [[977, 73]]], dtype=int32) )  Any help is appreciated! Thanks in advance!

Answer: You can use sorted on your tuple and specify the item using the key parameter First sort ip = sorted(ip, key=lambda x: x[0][0][0]) print(ip) [array([[[ 17, 3]], [[ 16, 4]], [[ 0, 4]], [[ 0, 49]], [[197, 49]], [[197, 8]], [[ 84, 4]], [[ 83, 3]]]), array([[[ 50, 73]], [[ 50, 107]], [[ 55, 108]], [[ 55, 121]], [[978, 87]], [[977, 86]], [[977, 73]]]), array([[[ 669, 3]], [[ 668, 4]], [[ 667, 4]], [[1033, 71]], [[1035, 69]], [[1035, 4]], [[ 848, 4]], [[ 847, 3]], [[ 813, 3]], [[ 718, 4]], [[ 717, 3]]])]  And the second sort ip = sorted(ip, key=lambda x: x[0][0][1]) print(ip) [array([[[ 17, 3]], [[ 16, 4]], [[ 0, 4]], [[ 0, 49]], [[197, 49]], [[197, 8]], [[ 84, 4]], [[ 83, 3]]]), array([[[ 669, 3]], [[ 668, 4]], [[ 667, 4]], [[1033, 71]], [[1035, 69]], [[1035, 4]], [[ 848, 4]], [[ 847, 3]], [[ 813, 3]], [[ 718, 4]], [[ 717, 3]]]) array([[[ 50, 73]], [[ 50, 107]], [[ 55, 108]], [[ 55, 121]], [[978, 87]], [[977, 86]], [[977, 73]]])]  If you want it back as tuple just do tuple(ip)
Question: I made a variable annotation of a local variable: def a(): x: int = 1  But the annotations dict is empty: > a.__annotations__ {}  Same when I use inspect.get_annotations() or typing.get_type_hints(). How can I access the annotation? Is this feature implemented at all? I'm using Python 3.10.7.

Answer: OK, I found a way. As long as a function has a fqdn, its source code can be parsed and local annotations show there. import ast, inspect def a(): k: int = 1 print(ast.dump(ast.parse(inspect.getsource(a))))  gives  Module(body=[FunctionDef(name='a', args=arguments(posonlyargs=[], args=[], kwonlyargs=[], kw_defaults=[], defaults=[]), body=[AnnAssign(target=Name(id='k', ctx=Store()), annotation=Name(id='int', ctx=Load()), value=Constant(value=1), simple=1)], decorator_list=[])], type_ignores=[])  This is a pretty hacky way, but works for me.
Question: I had an issue when I installed Yapf this way: environment.systemPackages = with pkgs; [ (python311.withPackages(ps: with ps; [ toml python-lsp-server pyls-isort flake8 ])) pkgs.yapf ];  This gave me the error:  $ yapf autoapp.py yapf: toml package is needed for using pyproject.toml as a configuration file  And I solved when I did: environment.systemPackages = with pkgs; [ (python311.withPackages(ps: with ps; [ toml python-lsp-server pyls-isort flake8 yapf ])) ];  Why was the first configuration giving me an installed version of yapf that couldn't import toml?

Answer: These are the same package - you can see this by checking the source links from the package search page. Adding it to withPackages links the Python package with the interpreter, making it possible to run things like python -m yapf … or import yapf within the Python REPL. If you simply list pkgs.yapf at the top level, the package isn't linked to the Pyython interpreter, and so only things like man pages and executables are available in the resulting environment.
Question: I want to use ChromeDriver Performance Log to monitor network traffic. I'm looking for some usage like this: capabilities = DesiredCapabilities.CHROME # enable performance log capabilities['loggingPrefs'] = {"performance","all"} self.driver = webdriver.Chrome( desired_capabilities=capabilities ) # get performance log logs = driver.get_log("performance")  Since my version of selenium 4.10.0 desired_capabilities property is Deprecated. webdriver.Chrome(service=service, options=opts)  This option is allowed but I could not use desired_capabilities。 I want to know how to setup desired_capabilities to Chrome or how to open Performance Log in order to trace traffic details.

Answer: DesiredCapabilities earlier deprecated, is now completely removed in Selenium v4.10.  Solution You have to use an instance of ChromeOptions and the set_capability() method as follows: from selenium.webdriver.chrome.options import Options options = Options() options.set_capability('goog:loggingPrefs', {'performance': 'ALL'}) # other configurations driver = Chrome(service=service, options=options)
Question: This question might already have an answer, so please guide me to one if you know any. I couldn't find one myself, though this question feels like a common one. So, consider the following pattern: arr = [None] * n for i in range(n): # do some computations # ... # even more computations arr[i] = MyClass(some_computed_value)  So far so good, I tend to use this pattern from time to time. Now, let us be thorough in our attempt to provide all the code with type annotations. The problem is that we preallocate our array with Nones, so it has the type list[None]. But we want it to be list[MyClass]. How do we proceed? The most straightforward solution is making it optional: arr: list[Optional[MyClass]] = [None] * n  This solves the type checker issue, but now it's our issue since that Optional prohibits us from performing even basic operations on the result arr[0].my_method() # error: NoneType has no attribute "my_method"  Long story short, I end up with the following pattern: arr_: Any = [None] * n for i in range(n): # ... arr_[i] = MyClass(some_computed_value) arr = typing.cast(list[MyClass], arr_)  This is ugly, inconvenient, barely readable and boilerplate. What do you do?

Answer: You can lie to your type checker when you first initialize arr to "trick" it that arr never contains None entries. For the purposes of static type checking, arr will be a list[MyClass], even though it briefly contains None entries at runtime. You of course assume responsibility for making sure that this assumption plays out. For example, from typing import cast, reveal_type n = 1000 arr: list[int] = [cast(int, None)] * n # or alternatively arr: list[int] = [None] * n # type: ignore for i in range(n): arr[i] = i reveal_type(arr) reveal_type(arr[0])  passes type checking with both mypy and pyright, and outputs # mypy tmp.py:9: note: Revealed type is "builtins.list[builtins.int]" tmp.py:10: note: Revealed type is "builtins.int" Success: no issues found in 1 source file # pyright file.py file.py:9:13 - information: Type of "arr" is "list[int]" file.py:10:13 - information: Type of "arr[0]" is "int" 0 errors, 0 warnings, 2 informations
Question: I have some whole numbers stored in np.float64 arrays and scalars, which I want to convert to native Python int. This is my attempt: import numpy as np a = np.array([1, 2, 3], dtype=np.float64) b = np.float64(4) def float_to_int(x): x_object = x.astype(object) return np.floor(x_object) # Array inputs are converted to int print(type(float_to_int(a)[0])) # >> <class 'int'> # Scalar inputs are left as np.float64 print(type(float_to_int(b))) # >> <class 'numpy.float64'>  There are 3 things I don't understand here:  Why is the type casting different for scalars and arrays? Why did np.floor() do type casting at all (for array inputs)? How can I reliably cast np.float64 to int for scalars and arrays? 

Answer: I believe since Numpy and python data types are related but inherently different, you would have to explicitly convert it to python data type. One way to do it would be: a = a.astype(np.int64).tolist() b = int(b)  or alternatively a = a.astype(np.int64).astype(object) b = b.astype(np.int64).astype(object)  When you convert numpy array to object data type, it internally stores it as python objects. The object data type is flexible and inferred during conversion.
Question: Given a base exception type: class MyModuleError(Exception): pass  Suppose we have code that explicitly raises it, using exception chaining: def foo(): try: #some code except (ZeroDivisionError, OSError) as e: raise MyModuleError from e  Now, in the calling code... try: foo() except MyModuleError as e: # Now what?  how can I idiomatically write the except clause, so that the exception handling depends on the __cause__ (chained exception)? I thought of these approaches: a) using type(e) like:  # filter here t=type(e.__cause__) if t is ZeroDivisionError: doStuff() elif t is OSError: doOtherStuff() else: raise  b) using isinstance() like:  # filter here if isinstance(e.__cause__, ZeroDivisionError): doStuff() elif isinstance(e.__cause__, OSError): doOtherStuff() else: raise  c) re-raising like:  # filter here try: raise e.__cause__ except ZeroDivisionError: doStuff() except OSError: doOtherStuff() except: raise e #which should be the "outer" exception 

Answer: The re-raising is not recommended. In general, it is not idiomatic to raise something deliberately so that it can be immediately caught1. It's also less performant (exception handling can involve quite a bit of overhead when the exception is actually raised), and creates a reference cycle between the exception object and the current stack frame, which in the reference implementation of Python will leak memory when the auxiliary garbage collector is disabled. (This is why exception names created with as are explicitly deleted after the except block in 3.x.) On general principle, isinstance is the preferred means of type-checking rather than comparing type results directly, because isinstance automatically takes subtyping into account. The normal functionality of except accounts for subtyping (e.g. except IOError: will catch a FileNotFoundError, which is normally desirable); it stands to reason that, in any normal circumstance, type-checking of the chained exception should do this as well. There is no explicit built-in functionality for this; therefore, approach b) is recommended here. 1Yes, for loops are internally implemented this way, using StopIteration. That's an implementation detail, and user code isn't intended to look like it's doing such things.
Question: I would like to have a custom ruamel.yaml dumper that uses Literal style for all multiline strings and the default style otherwise. For example: import sys import ruamel.yaml data = {"a": "hello", "b": "hello\nthere\nworld"} print("Default style") yaml = ruamel.yaml.YAML() yaml.dump(data, sys.stdout) print() print("style='|'") yaml = ruamel.yaml.YAML() yaml.default_style = "|" yaml.dump(data, sys.stdout)  This produces: Default style a: hello b: "hello\nthere\nworld" style='|' "a": |- hello "b": |- hello there world  My desired output is: a: hello b: |- hello there world 

Answer: There are multiple ways to achieve what you want. If you have control over building up the data structure, it is often easiest to add a LiteralScalarString if appropriate: import sys import ruamel.yaml def lim(s): # literal if multi-line if '\n' in s: return ruamel.yaml.scalarstring.LiteralScalarString(s) return s data = {'a': lim('hello'), 'b': lim('hello\nthere\nworld')} yaml = ruamel.yaml.YAML() yaml.dump(data, sys.stdout)  which gives: a: hello b: |- hello there world  This gives you easy fine control over what gets dumped as literal style. If you don't add all the data individually (but e.g. read them from a JSON file), you can walk over your data structure after it is fully constructed and update it in place: import sys import ruamel.yaml def tmtl(d): """translate multi-line to literal, only acts on dict values and sequence items, not on keys """ if isinstance(d, dict): for k, v in d.items(): if isinstance(v, str) and '\n' in v: d[k] = ruamel.yaml.scalarstring.LiteralScalarString(v) else: tmtl(v) elif isinstance(d, list): for idx, item in enumerate(d): if isinstance(item, str) and '\n' in item: d[idx] = ruamel.yaml.scalarstring.LiteralScalarString(item) data = {'a': 'hello', 'b': 'hello\nthere\nworld'} tmtl(data) yaml = ruamel.yaml.YAML() yaml.dump(data, sys.stdout)  which gives: a: hello b: |- hello there world  If you cannot update your data, you could rewrite tmtl in the program above so it builds a new data structure and returns that, but at that point it is IMO easier to change the representer: import sys import ruamel.yaml CKS = ruamel.yaml.comments.CommentedKeySeq # so you can have sequences as keys in a mapping class MyRepresenter(ruamel.yaml.representer.RoundTripRepresenter): def represent_str(self, s): if '\n' in s: return self.represent_scalar('tag:yaml.org,2002:str', s, style='|') return self.represent_scalar('tag:yaml.org,2002:str', s) MyRepresenter.add_representer(str, MyRepresenter.represent_str) data = {'a': 'hello', 'b': 'hello\nthere\nworld', CKS((1, 2)): ['nested works\nas well\n\n']} yaml = ruamel.yaml.YAML() yaml.Representer = MyRepresenter yaml.dump(data, sys.stdout)  which gives: a: hello b: |- hello there world [1, 2]: - |+ nested works as well ...  As you can see the trailing newlines of the final literal style scalar automatically causes the chomping indicator to change from strip (-) to keep (+) and the explicit document end marker (...) to appear.
Question: I have .csv file like this: Str; Int; Flt A; 123; 0.1 B; 456; 0.2 C; 789; 0.3  I want to get DataFrame like this  Int; Flt A; 123; 0.1 B; 456; 0.2 C; 789; 0.3  I read csv like this df = pd.read_csv('data.csv', index_col=0, sep=";")  And the problem is that I can't use df.loc["A", "Int"] to get cell value. If I drop Str; from csv everything works fine. So the idea is to use first row as row names and first column as column names. I understand that first element can't be used both as col name and row name, is there any way to drop such ambigous value?

Answer: You have a whitespace issue. import io import pandas as pd fp = io.StringIO(''' Str; Int; Flt A; 123; 0.1 B; 456; 0.2 C; 789; 0.3 '''.strip()) df = pd.read_csv(fp, index_col=0, sep=";") print("Index: ", df.index) print("Columns: ", df.columns) # Now look at the whitespace: print(df.loc[' A',' Int'])  Yields: Index: Index([' A', ' B', ' C'], dtype='object', name='Str') Columns: Index([' Int', ' Flt'], dtype='object') 123  So when you get rid of "Str", it appears you're dealing with the whitespace issue. So instead do: df = pd.read_csv(fp, index_col=0, sep=";" , skipinitialspace=True) print(df.loc['A','Int'])
Question: I started profiling my application and I tested to following code: a = np.random.random((100000,)) def get_first_index(value, arr): firstIndex = np.argmax(arr > value) if firstIndex <= 0: raise Exception('No index found') return firstIndex for i in range(0, 1000): get_first_index(0.5, a)  It just returns me the first index of an element bigger than the given value. On my machine it takes around 0.01s for array size 50k and 1k calls. I was wondering what causes the slow down. My first suspect was np.argmax but I boiled it down to the boolean comparison arr > value. It spends 99% of the time creating the bool comparison. Is there any faster way I am not aware of? Test code for profiling: a = np.random.random((100000,)) def test_function(a, b): return a < b import cProfile, pstats profiler = cProfile.Profile() profiler.enable() for i in range(0, 1000): test_function(0.5, a) profiler.disable() stats = pstats.Stats(profiler).sort_stats('tottime') stats.print_stats() 

Answer: The reason your approach is slow is that all elements of arr are compared under all circumstances, even if the first element of arr is greater than value. While numpy does not have an API optimized for this kind of processing, Numba can be used instead, which can be easily implemented as follows. import timeit import numba import numpy as np np.random.seed(0) a = np.random.random((100000,)) def get_first_index(value, arr): firstIndex = np.argmax(arr > value) if firstIndex <= 0: raise Exception("No index found") return firstIndex @numba.njit("i8(f8,f8[:])", cache=True) def get_first_index2(value, arr): for i in range(len(arr)): if arr[i] > value: return i raise Exception("No index found") x = 0.9999 print(timeit.timeit(lambda: get_first_index(x, a), number=1000)) print(timeit.timeit(lambda: get_first_index2(x, a), number=1000))  0.01520979999999994 0.001343500000000053  As a side note, if arr is a sorted array, np.searchsorted is even faster.
Question: I have searched a lot for the solution but still struggling with this problem. I'm trying to connect to a SQL Server instance running on 127.0.0.1:1433. However, I'm getting a sqlalchemy.exc.DBAPIError with the following error message: sqlalchemy.exc.DBAPIError: (pyodbc.Error) ('01000', "[01000] [unixODBC][Driver Manager]Can't open lib 'ODBC Driver 17 for SQL Server' : file not found (0) (SQLDriverConnect)")  I think I need to install the ODBC driver, but I'm not sure if it needs to be installed on the SQL Server Docker image or on my local VM. If the answer is the Docker image, then I think my /etc/odbcinst.ini file is correctly configured as follows: [ODBC Driver 17 for SQL Server] Description=Microsoft ODBC Driver 17 for SQL Server Driver=/opt/microsoft/msodbcsql17/lib64/libmsodbcsql-17.10.so.2.1 UsageCount=1  But if the ODBC driver needs to be installed on my local VM, then my /etc/odbcinst.ini file is empty. Here's the Python code I used to connect to the SQL Server instance: from sqlalchemy import create_engine server = "127.0.0.1,1433" user = "sa" password = "Pass@12345" db_name = "test_database" engine = create_engine(f'mssql+pyodbc://{user}:{password}@{server}/{db_name}?driver=ODBC Driver 17 for SQL Server') connection = engine.connect() print("connected")  Another question is what should i do if there is @ in password?  sqlserver: sqlserver:2022-latest docker image, runs on 127.0.0.1:1433 os: Ubuntu 22.04 python: 3.10.6 sqlalchemy: 2.0.16  Any help would be greatly appreciated. Thanks!

Answer: Solved by these steps: 1- install ODBC driver on local machine using this script for ubuntu official doc: if ! [[ "18.04 20.04 22.04 22.10" == *"$(lsb_release -rs)"* ]]; then echo "Ubuntu $(lsb_release -rs) is not currently supported."; exit; fi sudo su curl https://packages.microsoft.com/keys/microsoft.asc | apt-key add - curl https://packages.microsoft.com/config/ubuntu/$(lsb_release -rs)/prod.list > /etc/apt/sources.list.d/mssql-release.list exit sudo apt-get update sudo ACCEPT_EULA=Y apt-get install -y msodbcsql18 # optional: for bcp and sqlcmd sudo ACCEPT_EULA=Y apt-get install -y mssql-tools18 echo 'export PATH="$PATH:/opt/mssql-tools18/bin"' >> ~/.bashrc source ~/.bashrc # optional: for unixODBC development headers sudo apt-get install -y unixodbc-dev   Note: DNS filtering may cause issues during installation  2- Edit script as bellow to if password contains char @ from sqlalchemy import create_engine from urllib.parse import quote_plus server = "127.0.0.1:1433" user = "sa" password = "Pass@12345" db_name = "test_database" dsn = "ODBC Driver 18 for SQL Server" engine = create_engine(f"mssql+pyodbc://{user}:%s@{server}/{db_name}?TrustServerCertificate=yes&driver={dsn}" % quote_plus(password)) connection = engine.connect() print("connected")
Question: The following is valid SQL for PostgreSQL. INSERT INTO schema.table (id, letter) VALUES (1, 'a'), (2, 'b'), ...  I would like to execute a similar, parameterized statement using SQLAlchemy. I am using SQLAlchemy 1.4* Connection.execute(), but I do not have access to the table mapper class (ORM model). The following does not work, but does demonstrate what I am trying to achieve. statement: str = """ INSERT INTO schema.table (id, letter) VALUES :values """ values: Tuple[Tuple[int, str], ...] = ( (1, "a"), (2, "b"), ) with engine.connect() as connection: connection.execute( sqlalchemy.text(statement), {"values": values}, )  In this exact example, values is a tuple; it gets bound as a tuple of tuples, which does not fit what I am trying to achieve ("INSERT has more expressions than target columns"). Likewise, using a list of tuples creates a SQL ARRAY.  Q: Why don't you just use an f-string or "...".format(...)?   A: I've read that this is poor practice.  Question: How can I properly "unpack" the values parameters? Alternatively, what is the best way to achieve what I am seeking (without using the Table ORM class)?  * Yes, I am aware that SQLAlchemy 1.4 is deprecated at the time of this question's posting.

Answer: You can use the following, you were almost there, you just had to change the insert query and make the binding as a list of dicts. As seen in the docs and slightly modified for your use-case. statement: str = "INSERT INTO schema.table (id, letter) VALUES (:id, :letter)" values = ( (1, "a"), (2, "b"), ) values = [{'id': id, 'letter': letter} for id, letter in values] with engine.connect() as connection: connection.execute(text(statement), values)
Question: I have a column of Numpy arrays in Pandas, something like:  col1 col2 col3 0 1 a None 1 2 b [2, 4] 2 3 c None  The [2, 4] is really np.array([2, 4]). Now I need to impute the missing values, and I have a list of arrays for that. For example: vals_to_impute = [np.array([1, 2]), np.array([1, 4])]  I tried: mask = col3.isna() df.loc[mask, "col3"] = vals_to_impute  This results in error: ValueError: Must have equal len keys and value when setting with an ndarray  I tried converting to Numpy array, extracting column etc., nothing worked. Is it actually possible to set this in a vectorized operation, or do I have to do a manual loop?

Answer: I managed to do it using pd.Series instead of list. I also had to input an index to this Series so that the insertion is correct. Maybe it can be done easier. df = pd.DataFrame({ "col1": [1, 2, 3], "col2": ["a", "b", "c"], "col3": [None, np.array([2, 4]), None] }) mask = df["col3"].isna() vals_to_impute = pd.Series( [np.array([1, 2]), np.array([1, 4])], index=mask[mask].index ) df.loc[mask, "col3"] = vals_to_impute print(df)  Output:  col1 col2 col3 0 1 a [1, 2] 1 2 b [2, 4] 2 3 c [1, 4]
Question: Usually, I use % string formatting. But, as I discovered f-string is the new way of string formatting and it is faster as well, I want to use it.  But, I am facing a problem when formatting a float into an integer using f-string. Following is what I have tried: Using % string formatting '%5d'%1223 # yields --> ' 1223' '%5d'%1223.555 # yields --> ' 1223'  Using f-string formatting f'{1223:5d}' # yields --> ' 1223' ==> correct f'{1223.555:5d}' # gives error # error is "ValueError: Unknown format code 'd' for object of type 'float'"  Am I missing something?

Answer: The reason for this error is that the format specifier d is specifically for formatting integers, not floats.You can use the general format specifier f instead. f'{int(1223.555):5d}'