eshangj/stackoverflow_q_and_a_sample huggingface dataset:
Question: I'm developing Azure Functions using Python 3.10.10 on my machine, deploying the Function through Azure DevOps which is building the artifact using Python 3.6.8, and the Python Version shown for the Function App host is 3.8. There was a recent update of Azure Functions Runtime which deprecated Python 3.6. (see breaking changes here). How does python version affect Azure Functions? How do we keep the versions aligned?

Answer:  Alignment  Always keep in Azure DevOps a version of python venv that matches the App host and also keep the same dependencies within a requirements.txt file so that you don't have conflicts from different libraries. On your local you should also have a python venv that matches the same version of the host. I would suggest to downgrade from your current 3.10.10 to 3.8.x just to avoid having conflicts.  Python version  Python version per se shouldn't generally affect functionality (unless there are big changes between minor versions), it's usually the library dependencies that are breaking the functionalities (deprecation and implementations of methods in different way)
Question: I'm trying to write a search based on tuples with the Django ORM syntax. The final sql statement should look something like: SELECT * FROM mytable WHERE (field_a,field_b) IN ((1,2),(3,4));  I know I can achieve this in django using the extra keyword: MyModel.objects.extra( where=["(field_a, field_b) IN %s"], params=[((1,2),(3,4))] )  but the "extra" keyword will be deprecated at some point in django so I'd like a pure ORM/django solution. Searching the web, I found https://code.djangoproject.com/ticket/33015 and the comment from Simon Charette, something like the snippet below could be OK, but I can't get it to work. from django.db.models import Func, lookups class ExpressionTuple(Func): template = '(%(expressions)s)' arg_joiner = "," MyModel.objects.filter(lookups.In( ExpressionTuple('field_a', 'field_b'), ((1,2),(3,4)), ))  I'm using Django 3.2 but I don't expect Django 4.x to do a big difference here. My db backend is posgresql in case it matters.

Answer: For reference and inspired from akshay-jain proposal, I managed to write something that works: from django.db.models import Func,Value def ValueTuple(items): return tuple(Value(i) for i in items) class Tuple(Func): function = '' qs = ( MyModel.objects .alias(a=Tuple('field_a', 'field_b')) .filter(a__in=ValueTuple([(1, 2), (3, 4)]) )  It does produces a sql query like SELECT * FROM table WHERE (field_a,field_b) IN ((1,2),(3,4));  And can be extended to more fields than just two. I didn't do any benchmarks to compare it to Q objects filtering though.
Question: I installed TensorFlow under WSL 2, Ubuntu 22.04 (Jammy Jellyfish), I followed the instructions in Install TensorFlow with pip. *I also installed Nvidia drivers for Windows and in my other WSL 2, I use GPU-supported simulation program. Everything seemed OK. I didn't get any error message during installation, but when I imported TensorFlow in Python 3, I got this error: 2023-02-12 14:49:58.544771: W tensorflow/compiler/xla/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libnvinfer.so.7'; dlerror: libnvrtc.so.11.0: cannot open shared object file: No such file or directory 2023-02-12 14:49:58.544845: W tensorflow/compiler/xla/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libnvinfer_plugin.so.7'; dlerror: libnvinfer_plugin.so.7: cannot open shared object file: No such file or directory 2023-02-12 14:49:58.544874: W tensorflow/compiler/tf2tensorrt/utils/py_utils.cc:38] TF-TRT Warning: Cannot dlopen some TensorRT libraries. If you would like to use Nvidia GPU with TensorRT, please make sure the missing libraries mentioned above are installed properly.  I searched my libnvinfer_plugin.so.7 files: sudo find / -name libnvinfer.so.7 2> /dev/null  and I found them in this directory: cat /usr/lib/x86_64-linux-gnu/libnvinfer.so.7  and I added this directory to LD_LIBRARY_PATH like in Could not load dynamic library 'libnvinfer.so.7', but nothing changed. Still TensorFlow is working, but I can't use the GPU. nvidia-smi: +-----------------------------------------------------------------------------+ | NVIDIA-SMI 515.65.01 Driver Version: 516.94 CUDA Version: 11.7 | |-------------------------------+----------------------+----------------------+ | GPU Name Persistence-M| Bus-Id Disp.A | Volatile Uncorr. ECC | | Fan Temp Perf Pwr:Usage/Cap| Memory-Usage | GPU-Util Compute M. | | | | MIG M. | |===============================+======================+======================| | 0 NVIDIA GeForce ... On | 00000000:01:00.0 Off | N/A | | N/A 43C P0 22W / N/A | 0MiB / 6144MiB | 0% Default | | | | N/A | +-------------------------------+----------------------+----------------------+ +-----------------------------------------------------------------------------+ | Processes: | | GPU GI CI PID Type Process name GPU Memory | | ID ID Usage | |=============================================================================| | No running processes found | +-----------------------------------------------------------------------------+  nvcc--version: nvcc: NVIDIA (R) Cuda compiler driver Copyright (c) 2005-2023 NVIDIA Corporation Built on Fri_Jan__6_16:45:21_PST_2023 Cuda compilation tools, release 12.0, V12.0.140 Build cuda_12.0.r12.0/compiler.32267302_0  *The TensorFlow version is: 2.11.0 So, how can I fix this problem?

Answer: I changed version and the problem was solved: pip install --upgrade tensorflow==2.8  Note: When I use v2.10, I get the same error message. v2.8 is stable now.
Question: I need to process my input file values, turning them into a comma-separated string (instead of white space) in order to pass them to a CLI program. To do this, I want to run the input files through a Python function. How can I reference the input files of a rule in the params section of the same rule? This is what I've tried, but it doesn't work: rule a: input: foo="a.txt", bar=expand({build}.txt,build=config["build"]), output: baz=result.txt, params: joined_bar=lambda w: ",".join(input.bar), # this doesn't work shell: """ qux --comma-separated-files {params.joined_bar} \ --foo {input.foo} \ >{output.baz} """  It fails with: InputFunctionException: AttributeError: 'builtin_function_or_method' object has no attribute 'bar'  Potentially related but (over-)complicated questions:  How to define parameters for a snakemake rule with expand input  Is Snakemake params function evaluated before input file existence?

Answer: Turns out I need to explicitly add input to the lambda w: part: rule a: input: foo="a.txt", bar=expand({build}.txt,build=config["build"]), output: baz=result.txt, params: joined_bar=lambda w, input: ",".join(input.bar), # ', input' was added shell: """ qux --comma-separated-files {params.joined_bar} \ --foo {input.foo} \ >{output.baz} """  Interestingly, I found that one needs to use input in the lambda w, input. In my testing, lambda w, i did not work. And alternative is to refer to the rule input in the standard way: rules.a.input.bar: rule a: input: foo="a.txt", bar=expand({build}.txt,build=config["build"]), output: baz=result.txt, params: joined_bar=lambda w: ",".join(rules.a.input.bar), # 'rules.a.' was added shell: """ qux --comma-separated-files {params.joined_bar} \ --foo {input.foo} \ >{output.baz} """  Also see http://biolearnr.blogspot.com/2017/11/snakemake-using-inputoutput-values-in.html for a discussion.
Question: I'm migrating an old python project to the new pyproject.toml based system and am having trouble with getting files that are required by tests to install. Inside the pyproject.toml I have: [tool.setuptools] package-data = {"my_pkg_name" = ["tests/*.sdf", "tests/*.urdf", "tests/*.xml", "tests/meshes/*.obj"]} [build-system] requires = ["setuptools>=43.0.0", "wheel"] build-backend = "setuptools.build_meta"  The tests that are run with pytest require the files described under package-data. After I build and install the build, the test files are not there. How do I get those files to be installed? How to include package data with setuptools/distutils? may be related, but things have changed, and I would rather not have to create a manifest file. The project structure looks something like: . ├── LICENSE.txt ├── pyproject.toml ├── README.md ├── src │ ├── my_pkg_name │ │ ├── __init__.py └── tests ├── ant.xml ├── humanoid.xml ├── __init__.py ├── kuka_iiwa.urdf ├── meshes │ ├── link_0.obj │ ├── link_1.obj │ ├── link_2.obj │ ├── link_3.obj │ ├── link_4.obj │ ├── link_5.obj │ ├── link_6.obj │ └── link_7.obj └── test_transform.py  The pyproject.toml has no specific package discovery related settings.

Answer: Looking more into this, specifically: https://setuptools.pypa.io/en/stable/userguide/datafiles.html#non-package-data-files my problem is that these files are not part of the default found packages which are just the ones under src. It is also not clear whether test files should be installed or not - many projects explicitly exclude test files from installation. To answer the original question of how to install the test files, the solution is to make tests a package (as I already have above), then specify to find it with setup tools like so: [tool.setuptools.packages.find] where = ["src", "tests"] [tool.setuptools.package-data] "*" = ["*.sdf", "*.urdf", "*.xml", "*.obj"]  I believe package-data requires setuptools >= 61 which you can specify like: [build-system] requires = ["setuptools>=61.0.0", "wheel"] build-backend = "setuptools.build_meta"  Further answering the deeper issue of installed tests failing due to not being able to find test files, the solution in my github actions workflow for doing automated testing on commits was to install in editable mode. Specifically:  - name: Install dependencies run: | python -m pip install --upgrade pip python -m pip install -e . python -m pip install flake8 pytest if [ -f requirements.txt ]; then pip install -r requirements.txt; fi  This is because in my tests, I refer to resource files based on a module's path: open(os.path.join(cfg.TEST_DIR, "simple_arm.sdf"))  in the cfg module: import os ROOT_DIR = os.path.abspath(os.path.join(os.path.dirname(__file__), '../../')) TEST_DIR = os.path.join(ROOT_DIR, 'tests')  EDIT: A better solution is to reference the test files based on the test script path rather than the package's module's path. This allows us to still test the non-editable installed project. To do this, I have instead TEST_DIR = os.path.dirname(__file__)  in each test script, and replace references to cfg.
Question: With the code: from lxml.etree import HTML, XML  I get the traceback: Traceback (most recent call last): File "/Users/username/code/project/lxml-test.py", line 3, in <module> from lxml.etree import HTML, XML ImportError: dlopen(/Users/username/.virtualenvs/project-venv/lib/python3.11/site-packages/lxml/etree.cpython-311-darwin.so, 0x0002): symbol not found in flat namespace '_xsltDocDefaultLoader'  I'm on a mac m1 chip. I installed libxml2 and libxslt via brew. I'm running python 3.11 inside of a virtualenv. What I've tried:  Uninstalling and re-installing lxml with pip, and tried several different versions. (4.7.1 & 4.8.0 didn't compile. All of the 4.9.0,1,2 versions give me the above error) Installing libxml2 and libxslt via brew and then reinstalling python-lxml. Installing python-lxml via conda (as suggested here)  EDIT: I posted this bug in lxml's bug report forum, and was notified that this is a highly-duplicated bug report of Missing wheel for macos with M1 Edit

Answer: I solved my problem by cloning lxml, building it, and installing it via pip install -e /path/to/lxml
Question: Consider the following code example: # %% import numpy from scipy.interpolate import interp2d, RegularGridInterpolator x = numpy.arange(9000) y = numpy.arange(9000) z = numpy.random.randint(-1000, high=1000, size=(9000, 9000)) f = interp2d(x, y, z, kind='linear', copy=False) f2 = RegularGridInterpolator((x, y), z, "linear") mx, my = np.meshgrid(x, y) M = np.stack([mx, my], axis=-1) # %% %timeit f(x, y) # %% %timeit f2(M)  It sets up some example interpolators using scipy.interpolate.interp2d and scipy.interpolate.RegularGridInterpolator. The output of the two cells above is  1.09 s ± 4.38 ms per loop (mean ± std. dev. of 7 runs, 1 loop each)  and  10 s ± 17.6 ms per loop (mean ± std. dev. of 7 runs, 1 loop each)  respectively. The RegularGridInterpolator is about 10 times slower than the interp2d. The problem is that interp2d has been marked as deprecated in scipy 1.10.0. And new code should use RegularGridInterpolator. This seems a bit strange to me since it would be such a bad replacement. Is there maybe a problem in my code example above? How can I speed this interpolation process up?

Answer: There is no problem with your code, it's probably a bug in scipy. I've reported it on github
Question: I have never used OPC-UA before, but now faced with a task where I have to pull data from a OPC-UA machine to push to a SQL database using python. I can handle the database part, but how to basically connect to the OPCUA server when I have only the following fields available?  IP address 192.168.38.94 Port 8080 Security policy: Basic256 Username: della_client Password: amorphous@#  Some tutorials I saw directly use a url, but is there any way to form the URL from these parameters, or should I ask the machine owners something more specific to be able to connect? I just want to be sure of what I need before I approach him. Related, how to use the same parameters in the application called UA-Expert to verify the connections as well? Is it possible? If it is relevant, I am using python 3.10 on Ubuntu 22.04.

Answer: You need to know which protocol is used. Then you can create the URLs by using the IP address as domain:  OPC UA binary: opc.tcp://ip:port https https://ip:port OPC UA WebSockets opc.wss://ip:port http http://ip:port (Deprecated in Version 1.03)  In your example this could be opc.tcp://192.168.38.94:8080 or https://192.168.38.94:8080 In most cases, the binary protocol is used. But the port 8080 is a typical http(s) port. The credential and the securityPolice are needed later in the connection process. And yes: You can test the URLs with the UaExpert. You can finde a step-by-step tutorial in the documention
Question: Using Python and Starlette or FastAPI, How can I know if the request is coming from the Swagger UI or anywhere else (Postman, Frontend app)? I tried to see if there's something in Request object which I can use: from fastapi import Request @app.get("/") async def root(request: Request): # request.client.host just returns some IP # request.headers doesn't contain any hint # request.scope ? request_from_swagger = request.hints_on_whether_request_is_coming_from_swagger_ui if request_from_swagger: return {"message": "Hello Swagger UI"} return {"message": "Hello World"}  I need to take some actions based of that. So is there anyway I can tell, whether the request is coming from the Swagger UI?

Answer: You could always use the referer header of the request: from fastapi import Request @app.get("/") async def root(request: Request): request_from_swagger = request.headers['referer'].endswith(app.docs_url) if request_from_swagger: return {"message": "Hello Swagger UI"} return {"message": "Hello World"}
Question: Hi I'm struggling to get Tensorflow V2.11 to find my eGPU (RTX 3060 Ti) I am currently on Windows 11 CUDA version is 12 I am currently downloading CUDA 11 as well as CUDnn as I've heard it is recommended I have tried the following code: import tensorflow as tf tf.config.list_physical_devices('GPU')  which outputs:  []  any help would be great

Answer: Tensorflow 2.11 is not supporting GPU on Windows machine. TensorFlow 2.10 was the last TensorFlow release that supported GPU on native-Windows. So you can try by installing Tensorflow 2.10 for the GPU setup. Also you need to install the specific version of CUDA and cuDNN for GPU support in your system which is CUDA 11.2 and cuDNN 8.1 for Tensorflow 2.10(Tensorflow>=2.5). Please check the Hardware/Software requirements as mentioned in the link and set the path to the bin directory after installing these software. Now follow the step by step instructions mentioned in the same link and verify the GPU setup using below code. python -c "import tensorflow as tf; print(tf.config.list_physical_devices('GPU'))"
Question: I have a snakemake pipeline where I need to do a small step of processing the data (applying a rolling average to a dataframe). I would like to write something like this: rule average_df: input: # script = , df_raw = "{sample}_raw.csv" params: window = 83 output: df_avg = "{sample}_avg.csv" shell: """ python import pandas as pd df=pd.read_csv("{input.df_raw}") df=df.rolling(window={params.window}, center=True, min_periods=1).mean() df.to_csv("{output.df_avg}") """  However it does not work. Do I have to create a python file with those 4 lines of code? The alternative that occurs to me is a bit cumbersome. It would be average_df.py import pandas as pd def average_df(i_path, o_path, window): df=pd.read_csv(path) df=df.rolling(window=window, center=True, min_periods=1).mean() df.to_csv(o_path) return None if __name__ == "__main__": import argparse parser = argparse.ArgumentParser(description='Description of your program') parser.add_argument('-i_path', '--input_path', help='csv file', required=True) parser.add_argument('-o_path', '--output_path', help='csv file ', required=True) parser.add_argument('-w', '--window', help='window for averaging', required=True) args = vars(parser.parse_args()) i_path = args['input_path'] o_path = args['output_path'] window = args['window'] average_df(i_path, o_path, window)  And then have the snakemake rule like this: rule average_df: input: script = average_df.py, df_raw = "{sample}_raw.csv" params: window = 83 output: df_avg = "{sample}_avg.csv" shell: """ python average_df.py --input_path {input.df_raw} --ouput_path {output.df_avg} -window {params.window} """  Is there a smarter or more efficient way to do this? That would be great! Looking forward to your input!

Answer: This can be achieved via run directive: rule average_df: input: # script = , df_raw = "{sample}_raw.csv" params: window = 83 output: df_avg = "{sample}_avg.csv" run: import pandas as pd df=pd.read_csv(input.df_raw) df=df.rolling(window=params.window, center=True, min_periods=1).mean() df.to_csv(output.df_avg)  Note that all snakemake objects are available directly via input, output, params, etc.
Question: Does anyone know how to parse YYYY Week into a date column in Polars? I have tried this code but it throws an error. import polars as pl pl.DataFrame({ "week": [201901, 201902, 201903, 201942, 201943, 201944] }).with_columns(pl.col("week").cast(pl.String).str.to_date("%Y%U").alias("date"))  InvalidOperationError: conversion from `str` to `date` failed in column 'week' for 6 out of 6 values: ["201901", "201902", … "201944"] 

Answer: This seems like a bug (although one with the underlying rust package chrono rather than polars itself). I tried using base python's strptime and it ignores the %U and just gives the first of the year for all cases so you can either do string manipulation and math like this (assuming you don't need an exact response) pl.DataFrame({ "week": [201901, 201902, 201903, 201942, 201943, 201944] }) \ .with_columns(pl.col('week').cast(pl.Utf8)) \ .with_columns([pl.col('week').str.slice(0,4).cast(pl.Int32).alias('year'), pl.col('week').str.slice(4,2).cast(pl.Int32).alias('week')]) \ .select(pl.date(pl.col('year'),1,1) + pl.duration(days=(pl.col('week')-1)*7).alias('date'))  If you look at the definition of %U, it's supposed to be based the xth Sunday of the year whereas my math is just multiplying by 7. Another approach is to make a df of dates, then make the strftime of them and then join the dfs. So that might be like this: dfdates=pl.DataFrame({'date':pl.date_range(datetime(2019,1,1), datetime(2019,12,31),'1d').cast(pl.Date())}) \ .with_columns(pl.col('date').dt.strftime("%Y%U").alias('week')) \ .groupby('week').agg(pl.col('date').min())  And then joining it with what you have pl.DataFrame({ "week": [201901, 201902, 201903, 201942, 201943, 201944] }).with_columns(pl.col('week').cast(pl.Utf8())).join(dfdates, on='week') shape: (6, 2) ┌────────┬────────────┐ │ week ┆ date │ │ --- ┆ --- │ │ str ┆ date │ ╞════════╪════════════╡ │ 201903 ┆ 2019-01-20 │ │ 201944 ┆ 2019-11-03 │ │ 201902 ┆ 2019-01-13 │ │ 201943 ┆ 2019-10-27 │ │ 201942 ┆ 2019-10-20 │ │ 201901 ┆ 2019-01-06 │ └────────┴────────────┘
Question: In the python code below, the glm model specification does not include the third power in the in model1 but it does in model2: model1 = glm(formula="wage ~ workhours + workhours**3 + C(gender)", data=df, family=sm.families.Gaussian()) model2 = glm(formula="wage ~ workhours + np.power(workhours, 3) + C(gender)", data=df, family=sm.families.Gaussian())  Is this a bug? According to the documentation **x raises something to the power 3.

Answer: ** in a formula is treated as a formula operator, not as regular exponentiation. (This is similar to how ^ works in an R formula.) (a+b+c+d)**3 means that the model should include a, b, c, d, and all interactions between these variables up to 3rd order. workhours**3 means that the model should include workhours and all interactions between... just workhours... up to 3rd order... but there are no such interaction terms, so it's equivalent to just workhours. In contrast, np.power(workhours, 3) is treated as Python code, and computes the power you wanted. statsmodels uses patsy for formula handling, so for full details on the formula language, you can check the patsy docs.
Question: I am working on google colab with the segmentation_models library. It worked perfectly the first week using it, but now it seems that I can't import the library anymore. Here is the error message, when I execute import segmentation_models as sm : --------------------------------------------------------------------------- AttributeError Traceback (most recent call last) <ipython-input-3-6f48ce46383f> in <module> 1 import tensorflow as tf ----> 2 import segmentation_models as sm 3 frames /usr/local/lib/python3.8/dist-packages/efficientnet/__init__.py in init_keras_custom_objects() 69 } 70 ---> 71 keras.utils.generic_utils.get_custom_objects().update(custom_objects) 72 73 AttributeError: module 'keras.utils.generic_utils' has no attribute 'get_custom_objects'  Colab uses tensorflow version 2.11.0. I did not find any information about this particular error message. Does anyone know where the problem may come from ?

Answer: Encountered the same issue sometimes. How I solved it:  open the file keras.py, change all the 'init_keras_custom_objects' to 'init_tfkeras_custom_objects'.  the location of the keras.py is in the error message. In your case, it should be in /usr/local/lib/python3.8/dist-packages/efficientnet/