eshangj/stackoverflow_q_and_a_sample huggingface dataset:
Question: I am not sure why I am getting this error although sometimes my code works fine! Excel file format cannot be determined, you must specify an engine manually. Here below is my code with steps: 1- list of columns of customers Id: customer_id = ["ID","customer_id","consumer_number","cus_id","client_ID"]  2- The code to find all xlsx files in a folder and read them: l = [] #use a list and concat later, faster than append in the loop for f in glob.glob("./*.xlsx"): df = pd.read_excel(f).reindex(columns=customer_id).dropna(how='all', axis=1) df.columns = ["ID"] # to have only one column once concat l.append(df) all_data = pd.concat(l, ignore_index=True) # concat all data  I added the engine openpyxl df = pd.read_excel(f, engine="openpyxl").reindex(columns = customer_id).dropna(how='all', axis=1) Now I got a different error: BadZipFile: File is not a zip file  pandas version: 1.3.0 python version: python3.9 os: MacOS is there a better way to read all xlsx files from a folder ?

Answer: Found it. When an excel file is opened for example by MS excel a hidden temporary file is created in the same directory: ~$datasheet.xlsx  So, when I run the code to read all the files from the folder it gives me the error: Excel file format cannot be determined, you must specify an engine manually.  When all files are closed and no hidden temporary files ~$filename.xlsx in the same directory the code works perfectly.
Question: I'm trying to run the code from this repository and I need to use Pytorch 1.4.0. I've installed the CPU only version of pytorch with pip install torch==1.4.0+cpu torchvision==0.5.0+cpu -f https://download.pytorch.org/whl/torch_stable.html. I ran the program by doing py -m train_Kfold_CV --device 0 --fold_id 10 --np_data_dir "C:\Users\username\OneDrive\Desktop\emadeldeen\AttnSleep\prepare_datasets\edf_20_npz" but I'm getting this error:  File "C:\Users\username\AppData\Local\Programs\Python\Python37\lib\runpy.py", line 193, in _run_module_as_main "__main__", mod_spec) File "C:\Users\username\AppData\Local\Programs\Python\Python37\lib\runpy.py", line 85, in _run_code exec(code, run_globals) File "C:\Users\username\OneDrive\Desktop\emadeldeen\AttnSleep\train_Kfold_CV.py", line 94, in <module> main(config, fold_id) File "C:\Users\username\OneDrive\Desktop\emadeldeen\AttnSleep\train_Kfold_CV.py", line 65, in main trainer.train() File "C:\Users\username\OneDrive\Desktop\emadeldeen\AttnSleep\base\base_trainer.py", line 66, in train result, epoch_outs, epoch_trgs = self._train_epoch(epoch, self.epochs) File "C:\Users\username\OneDrive\Desktop\emadeldeen\AttnSleep\trainer\trainer.py", line 49, in _train_epoch loss = self.criterion(output, target, self.class_weights) File "C:\Users\username\OneDrive\Desktop\emadeldeen\AttnSleep\model\loss.py", line 6, in weighted_CrossEntropyLoss cr = nn.CrossEntropyLoss(weight=torch.tensor(classes_weights).cuda()) File "C:\Users\username\AppData\Local\Programs\Python\Python37\lib\site-packages\torch\cuda\__init__.py", line 196, in _lazy_init _check_driver() File "C:\Users\username\AppData\Local\Programs\Python\Python37\lib\site-packages\torch\cuda\__init__.py", line 94, in _check_driver raise AssertionError("Torch not compiled with CUDA enabled") AssertionError: Torch not compiled with CUDA enabled  I've changed the number of GPU in the config to 0 and tried adding device = torch.device('cpu') at the begining of the program, but it's not doing anything. How can I fix this error? I'm using windows 10 with python 3.7.9 if it helps Thanks

Answer: You are using CPU only pytorch, but your code has statement like cr = nn.CrossEntropyLoss(weight=torch.tensor(classes_weights).cuda()) which is trying to move the tensor to GPU. To fix it, remove all the .cuda() operations.
Question: Minimum (not) working example  kernel = Conv2d(in_channels=1, out_channels=1, kernel_size=(3, 2)) data = torch.rand(1, 1, 100, 100).type(torch.complex64) kernel(data)  yields RuntimeError: "unfolded2d_copy" not implemented for 'ComplexDouble' for 64 and 128 bit complex numbers, while for 32 bit, i get RuntimeError: "copy_" not implemented for 'ComplexHalf'. Am I missing something, or is pytorch missing support for complex numbers?? note: I'm on macbook, using cpu only.

Answer: Currently (@ latest stable version - 1.9.0) Pytorch is missing support for such operations on complex tensors (which are a beta feature). See this feature request at Native implementation of convolution for complex numbers Splitting into convolution on real & image separately, though not ideal, is the way to go for now.
Question: I have a Lambda function that utilizes the AWS Python SDK to manage AWS CodeCommit repositories. I create the Lambda function using the CDK like so: from aws_cdk import aws_lambda as _lambda from aws_cdk.aws_lambda_python import PythonFunction service = PythonFunction( self, 'Svc', entry='./path/to', index='file.py', runtime=_lambda.Runtime.PYTHON_3_8, handler='handler', )  After deployment, I run the Lambda function, the following error occurs which is sent to CloudWatch logs: An error occurred (AccessDeniedException) when calling the GetRepository operation: User: arn:aws:iam::XXXXXXXXXXXX:user/XXXX is not authorized to perform: codecommit:GetRepository on resource: arn:aws:codecommit:us-east-1:XXXXXXXXXXXX:XXXX  How can I allow the Lambda function to call codecommit:GetRepository on any repository in my account?

Answer: Create an IAM Policy Statement and add it to your Function's role policy: from aws_cdk import aws_iam as iam service.add_to_role_policy(iam.PolicyStatement( effect=iam.Effect.ALLOW, actions=[ 'codecommit:*', ], resources=[ 'arn:aws:codecommit:us-east-1:XXXXXXXXXXXX:*', ], ))
Question: I want to do something like from typing import Callable def a(foo: Callable[[int], None]): foo(b=5)  This code works, but gives a warning Unexpected argument. Defining as def a(foo: Callable[[int], None]): foo(5)  works with no warnings as expected.  How can I pass in an expected argument as a kwarg into a function with the type checker not being angry at me?

Answer: The Callable docs say  There is no syntax to indicate optional or keyword arguments; such function types are rarely used as callback types.  However they also say  Callable[..., ReturnType] (literal ellipsis) can be used to type hint a callable taking any number of arguments and returning ReturnType  Applying here, that would be def a(foo: Callable[..., None]):  You will lose the int annotation, but it's either that, living with the warning or explicitly surpassing it.
Question: I have the following code: def extract_table_date(bucket_path: str) -> str: event_date = re.search(r"date=([^/]+)", bucket_path) return event_date.group(1)[0:10].replace("-", "")  mypy throws error on the last line:  Item "None" of "Optional[Match[str]]" has no attribute "group"  I think I can solve that by assigning a type to event_date, and I can: from typing import Match def extract_table_date(bucket_path: str) -> str: event_date: Match = re.search(r"date=([^/]+)", bucket_path) return event_date.group(1)[0:10].replace("-", "")  but mypy now throws another error on the first line of the function:  Incompatible types in assignment (expression has type "Optional[Match[Any]]", variable has type "Match[Any]")  I don't really know how to inform mypy that the result won't be optional but nonetheless I followed the advice at Optional types and the None type by adding an assert: from typing import Match def extract_table_date(bucket_path: str) -> str: assert bucket_path is not None event_date: Match = re.search(r"date=([^/]+)", bucket_path) return event_date.group(1)[0:10].replace("-", "")  but mypy still raises the same error. I try to fix by changing the type defined for event_date: from typing import Match, optional, Any def extract_table_date(bucket_path: str) -> str: assert bucket_path is not None event_date: Optional[Match[Any]] = re.search(r"date=([^/]+)", bucket_path) return event_date.group(1)[0:10].replace("-", "")  but (as expected) I'm now back to almost the same original error:  Item "None" of "Optional[Match[Any]]" has no attribute "group"  Any advice on how to fix this?

Answer: The thing that's Optional is event_date, because re.search is not guaranteed to return a match. mypy is warning you that this will raise an AttributeError if that's the case. You can tell it "no, I'm very confident that will not be the case" by doing an assert to that effect: def extract_table_date(bucket_path: str) -> str: event_date = re.search(r"date=([^/]+)", bucket_path) assert event_date is not None return event_date.group(1)[0:10].replace("-", "")  If you're wrong, this code will still raise an exception (AssertionError, because your assert will fail), but mypy will no longer error because there is now no way for event_date to be None when you access its group attribute. Note that there is no need to assert on bucket_path because it's already explicitly typed as str.
Question: When dataframe is like this,  a b c d 0 1.0 NaN 3.0 NaN 1 NaN 6.0 NaN 8.0 2 9.0 NaN NaN NaN 3 13.0 NaN 15.0 16.0  I want to remove rows that contain NaN in both b and d columns. So I want the result to be like this.  a b c d 1 NaN 6.0 NaN 8.0 3 13.0 NaN 15.0 16.0  In this situation I can't use df.dropna(thresh=2) because I don't want to erase row 1,  and if I use df.dropna(subset=['b', 'd']) then row 3 will be removed too. What should I do now?

Answer: dropna has an additional parameter, how: how{‘any’, ‘all’}, default ‘any’ Determine if row or column is removed from DataFrame, when we have at least one NA or all NA. ‘any’ : If any NA values are present, drop that row or column. ‘all’ : If all values are NA, drop that row or column.  If you set it to all, it will only drop the lines that are filled with NaN. In your case df.dropna(subset=['b', 'd'], how="all") would work.
Question: What's the best / most readable way to assert two querysets are equal? I've come up with a few solutions: # option 1 assert sorted(qs1.values_list("pk", flat=True)) == sorted(qs2.values_list("pk", flat=True)) # option 2 (need to assert length first because set might remove duplicates) assert len(qs1) == len(qs2) assert set(qs1) == set(qs2)  I know Django has a method django.test.TransactionTestCase.assertQuerysetEqual. Does pytest-django have something similar? I don't see it in the documentation.

Answer: It's there in the starting lines of the link that you suggested:  Assertions All of Django’s TestCase Assertions are available in pytest_django.asserts, e.g. from pytest_django.asserts import assertTemplateUsed  Similarly you can use, from pytest_django.asserts import assertQuerysetEqual
Question: I am trying to create an executable from a python script, using pyinstaller, and am getting the error seen in the subject line. The particulars:  python - version 3.9.2  pyinstaller - version 4  I am running on Debian Linux  I evoke pyinstaller as: pyinstaller --onefile pythonfile.py  When I looked to see what libpython*.so files were resident, I see libpython3.7*.so, and the error shows I need to install libpython3.9*so files. I have tried:  pip3 install PyInstaller (to load on pyinstaller) apt-get install python3-dev (as recommended in the pyinstaller error msg) apt-get install python-dev (as recommended in the pyinstaller error msg) apt-get upgrade apt-get update  but still get the error. How can I get the correct libpython*.so files loaded (i.e., 3.9)? TIA.

Answer: You need to generate the shared lib using: env PYTHON_CONFIGURE_OPTS="--enable-shared" pyenv install 3.9.2  I'm not sure whether 3.9.2 is working if not try 3.9.0 Official Document Here.
Question: I am trying to understand mypy a little better. For the following line of code: request_body: dict = {} request_body = request.get_json()  mypy returns an error: error: Incompatible types in assignment (expression has type "Optional[Any]", variable has type "Dict[Any, Any]")  What is the correct fix for this?

Answer: As you can see in the following code, taken from /wekzeug/wrappers/request.py, the function get_json doesn't always return a dictionary. I would suggest removing the type hinting from the variable, as it can be None or a dictionary. def get_json( self, force: bool = False, silent: bool = False, cache: bool = True ) -> t.Optional[t.Any]: """Parse :attr:`data` as JSON. If the mimetype does not indicate JSON (:mimetype:`application/json`, see :meth:`is_json`), this returns ``None``. If parsing fails, :meth:`on_json_loading_failed` is called and its return value is used as the return value. :param force: Ignore the mimetype and always try to parse JSON. :param silent: Silence parsing errors and return ``None`` instead. :param cache: Store the parsed JSON to return for subsequent calls. """ if cache and self._cached_json[silent] is not Ellipsis: return self._cached_json[silent] if not (force or self.is_json): return None data = self.get_data(cache=cache) try: rv = self.json_module.loads(data) except ValueError as e: if silent: rv = None if cache: normal_rv, _ = self._cached_json self._cached_json = (normal_rv, rv) else: rv = self.on_json_loading_failed(e) if cache: _, silent_rv = self._cached_json self._cached_json = (rv, silent_rv) else: if cache: self._cached_json = (rv, rv) return rv  This line specifically causes the method to return None: except ValueError as e: if silent: rv = None
Question: A few valid values for a tuple that I'm trying to annotate: ("foo", 1, 2) ("bar", 11) ("baz", 42, 31, 20, 0, -700, 44444, 12345, 1, 2, 3, 4, 5, 6, 7, 8, 9)  I was expecting this to work: my_tuple: Tuple[str, int, ...] # doesn't work!  ... but that throws error: Unexpected '...' Any way to annotate this structure?

Answer: In Python 3.11 I can now do the following: my_tuple: tuple[str, *tuple[int, ...]] = ("foo", 1, 2)  This was added in PEP 646  For Python pre-3.11 (courtesy of @FMeinicke) typing_extensions.Unpack can be used instead [1,2,3]. from typing_extensions import Unpack my_tuple: tuple[str, Unpack[tuple[int, ...]]] = ("foo", 1, 2)  though, note that there might still be issues with generic typed aliases pre-3.11, see https://github.com/python/typing_extensions/issues/103.
Question: I am working on one Django project. And I decided to write logic code in PostgreSQL instead of writing in Python. So, I created a stored procedure in PostgreSQL. For example, a stored procedure looks like this: create or replace procedure close_credit(id_loan int) language plpgsql as $$ begin update public.loan_loan set sum = 0 where id = id_loan; commit; end;$$  Then in settings.py, I made the following changes: DATABASES = { 'default': { 'ENGINE': 'django.db.backends.postgresql_psycopg2', 'NAME': 'pawnshop', 'USER': 'admin', 'PASSWORD': password.database_password, 'HOST': 'localhost', 'PORT': '', } }  So the question is, How can I call this stored procedure in views.py? p.s. Maybe it sounds like a dumb question, but I really couldn't find any solution in Django.

Answer: I'd recommend storing the procedure definition in a migration file. For example, in the directory myapp/migrations/sql.py: from django.db import migrations SQL = """ CREATE PROCEDURE close_credit(id_loan int) language plpgsql AS $$ BEGIN UPDATE public.loan_loan SET sum = 0 WHERE id = id_loan; COMMIT; END; $$ """ class Migration(migrations.Migration): dependencies = [ ('myapp', '0001_initial'), ] operations = [migrations.RunSQL(SQL)]  Note: you will need to replace myapp with the name of your application, and you will need to include only the most recent migration file for your app as a dependency. Now you can install the procedure using python3 manage.py migrate.  Once your procedure is defined in the database, you can call it using cursor.callproc: from django.db import connection def close_credit(id_loan): with connection.cursor() as cursor: cursor.callproc('close_credit', [id_loan])  All that being said, if your procedure is really as trivial as the example you provided, it would be better to use the ORM: Loan.objects.filter(id=id_loan).update(sum=0)
Question: I want to use f string formatting instead of print. However, I get these errors: Unterminated expression in f-string; missing close brace Expected ')' var="ab-c" f"{var.replace("-","")}text123"  I tried to use single quote f'' and also double brackets but neither of them worked. Any idea about how to fix this?

Answer: Before Python 3.12: For f"{var.replace("-","")}text123", Python parses f"{var.replace(" as a complete string, which you can see has an opening { and opening (, but then the string is terminated. It first expected a ) and eventually a }, hence the error you see. To fix it, Python allows ' or " to enclose a string, so use one for the f-string and the other for inside the string: f"{var.replace('-','')}text123"  or: f'{var.replace("-","")}text123'  Triple quotes can also be used if internally you have both ' and " f'''{var.replace("-",'')}text123'''  or: f"""{var.replace("-",'')}text123"""  As of Python 3.12: Some f-string limitations have been lifted and the OP's original code now works: Python 3.12.0 (tags/v3.12.0:0fb18b0, Oct 2 2023, 13:03:39) [MSC v.1935 64 bit (AMD64)] on win32 Type "help", "copyright", "credits" or "license" for more information. >>> var="ab-c" >>> f"{var.replace("-","")}text123" 'abctext123'  See What's New in Python 3.12, PEP 701: Syntactic formalization of f-strings.
Question: so in python you can split strings like this: string = "Hello world!" str1 , str2 = string.split(" ") print(str1);print(str2)  and it prints: Hello world!  How can i do the same in C++? This wasn't useful Parse (split) a string in C++ using string delimiter (standard C++) , i need them splited so i can acces them separatedly like print just str1 for example.

Answer: If your tokenizer is always a white space (" ") and you might not tokenize the string with other characters (e.g. s.split(',')), you can use string stream: #include <iostream> #include <string> #include <sstream> int main() { std::string my_string = " Hello world! "; std::string str1, str2; std::stringstream s(my_string); s>>str1>>str2; std::cout<<str1<<std::endl; std::cout<<str2<<std::endl; return 0; }  Keep in mind that this code is only suggested for whitespace tokens and might not be scalable if you have many tokens. Output: Hello World!
Question: PyCharm IDE. I want to use dbutils.widgets.get() in a module and than to import this module to databricks. I already tried with pip install databricks-client pip install databricks-utils and pip install DBUtils

Answer: The dbutils is available only as a part of the databricks-connect package. Its documentation contains detailed description on how to setup PyCharm to work with it. It also covers on how to use the dbutils. You may need to define following wrapper to be able to use dbutils locally and on Databricks: def get_dbutils(spark): from pyspark.dbutils import DBUtils return DBUtils(spark) get_dbutils().fs.cp('file:/home/user/data.csv', 'dbfs:/uploads')
Question: Let's say I want to initialize the below dataclass from dataclasses import dataclass @dataclass class Req: id: int description: str  I can of course do it in the following way: data = make_request() # gives me a dict with id and description as well as some other keys. # {"id": 123, "description": "hello", "data_a": "", ...} req = Req(data["id"], data["description"])  But, is it possible for me to do it with dictionary unpacking, given that the keys I need is always a subset of the dictionary? req = Req(**data) # TypeError: __init__() got an unexpected keyword argument 'data_a' 

Answer: Here's a solution that can be used generically for any class. It simply filters the input dictionary to exclude keys that aren't field names of the class with init==True: from dataclasses import dataclass, fields @dataclass class Req: id: int description: str def classFromArgs(className, argDict): fieldSet = {f.name for f in fields(className) if f.init} filteredArgDict = {k : v for k, v in argDict.items() if k in fieldSet} return className(**filteredArgDict) data = {"id": 123, "description": "hello", "data_a": ""} req = classFromArgs(Req, data) print(req)  Output: Req(id=123, description='hello')  UPDATE: Here's a variation on the strategy above which creates a utility class that caches dataclasses.fields for each dataclass that uses it (prompted by a comment by @rv.kvetch expressing performance concerns around duplicate processing of dataclasses.fields by multiple invocations for the same dataclass). from dataclasses import dataclass, fields class DataClassUnpack: classFieldCache = {} @classmethod def instantiate(cls, classToInstantiate, argDict): if classToInstantiate not in cls.classFieldCache: cls.classFieldCache[classToInstantiate] = {f.name for f in fields(classToInstantiate) if f.init} fieldSet = cls.classFieldCache[classToInstantiate] filteredArgDict = {k : v for k, v in argDict.items() if k in fieldSet} return classToInstantiate(**filteredArgDict) @dataclass class Req: id: int description: str req = DataClassUnpack.instantiate(Req, {"id": 123, "description": "hello", "data_a": ""}) print(req) req = DataClassUnpack.instantiate(Req, {"id": 456, "description": "goodbye", "data_a": "my", "data_b": "friend"}) print(req) @dataclass class Req2: id: int description: str data_a: str req2 = DataClassUnpack.instantiate(Req2, {"id": 123, "description": "hello", "data_a": "world"}) print(req2) print("\nHere's a peek at the internals of DataClassUnpack:") print(DataClassUnpack.classFieldCache)  Output: Req(id=123, description='hello') Req(id=456, description='goodbye') Req2(id=123, description='hello', data_a='world') Here's a peek at the internals of DataClassUnpack: {<class '__main__.Req'>: {'description', 'id'}, <class '__main__.Req2'>: {'description', 'data_a', 'id'}}
Question: In normal context, we just attach translation property to a variable like : this.name = this.$t('language.name');  But I want to specific it in a specific language sometime( ex: in French). Can we do something like this in vue.js ? this.name = this.$t('language.name', locale: fr); 

Answer: Using the old package kazupon/vue-i18n, the following should be possible: $t(key, locale)   When using the successor-package intlify/vue-i18n-next, the answer depends on if you are using Vue I18n's Legacy API or the newer Composition API:  Using Legacy API as described in the normal setup guide the usages of the t() function are stated here. This means you can still use the following call to translate a key to a specific locale (e.g. 'fr'): $t(key, locale)  Example: $t('message.key', 'fr')   Using the Composition API by calling createI18n() with the option legacy: false (as described here), the usages of the t() function are different as stated here. You can no longer pass the locale-string as the second parameter, but the locale can be handed over inside a TranslateOptions object. Unfortunately, there is no t(key,TranslateOptions) variant, but only the following variants: $t(key, plural, TranslateOptions) $t(key, defaultMsg, TranslateOptions) $t(key, interpolations, TranslateOptions)  So the easiest solution would be e.g.: $t('message.key', 1, { locale: 'fr' })