eshangj/stackoverflow_q_and_a_sample huggingface dataset:
Question: My goal is to deploy a Streamlit application to an AWS Serverless architecture. Streamlit does not appear to function properly without a Docker container, so the architecture would need to support containers. From various tutorials, EC2 is the most popular deployment option for Streamlit, which I have no interest in pursuing due to the server management aspect. AWS Lambda would be my preferred deployment option if viable. I see that Lambda can support containers, but I'm curious what the pros & cons of Lambda vs Fargate is for containerized apps. My question is: Is Lambda or Fargate better for a serverless deployment of a Streamlit web app?

Answer: AWS Lambda:  AWS Lambda can run containers, but those containers have to implement the Lambda runtime API. Lambda can't run any generic container. Lambda has a maximum run time (for processing a single request) of 15 minutes. Behind API gateway that maximum time is reduced to 60 seconds. Lambda isn't running 24/7. Your container would only be started up when a request comes in that the Lambda function needs to process. Given the nature of how Lambda works, something has to sit in front of Lambda to receive the web requests and route them to the AWS Lambda API. Most commonly this would be AWS API Gateway. So you would have to setup an AWS API Gateway deployment that understands how to route all of your apps API requests to your Lambda function(s). Alternatively you could put an Application Load Balancer in front of your Lambda function.  Fargate (or more appropriately titled "ECS services with Fargate deployments"):  Runs your container 24/7 like a more traditional web server.  Can run pretty much any container.  No time limit on the time to process a single request, although there is a maximum value of 4000 seconds (over 60 minutes) on the load balancer that you would typically use in this configuration.    So in general, it's much easier to take a third-party app or docker image and get it running in ECS/Fargate than it is to get it running in Lambda.
Question: The following lines are supposed to get the same result: import tensorflow as tf print(tf.random.truncated_normal(shape=[2],seed=1234)) print(tf.random.truncated_normal(shape=[2],seed=1234))  But I got: tf.Tensor([-0.12297685 -0.76935077], shape=(2,), dtype=tf.float32) tf.Tensor([0.37034193 1.3367208 ], shape=(2,), dtype=tf.float32)  Why?

Answer: This seems to be intentional, see the docs here. Specifically the "Examples" section. What you need is stateless_truncated_normal: print(tf.random.stateless_truncated_normal(shape=[2],seed=[1234, 1])) print(tf.random.stateless_truncated_normal(shape=[2],seed=[1234, 1]))  Gives me tf.Tensor([1.0721238 0.10303579], shape=(2,), dtype=float32) tf.Tensor([1.0721238 0.10303579], shape=(2,), dtype=float32)  Note: The seed needs to be two numbers here, I honestly don't know why (the docs don't say).
Question: Let's say I initialize a df and then I assign it to a dict 3 times, each one with a specific key. import pandas as pd df = pd.DataFrame({'A': [2, 2], 'B': [2, 2]}) dict = {} for i in range(3): dict_strat['Df {0}'.format(i)] = df  Alright, what I'm not understanding is that when I try to change value of one element in the dictionary, it changes all the others. For example: dict_strat['Df 0'].iloc[0, :] = 9  It not only changes the first df on the dict, it changes all of them. Why? How can I get rid of that?

Answer: The DataFrames are all shallow copies, meaning that mutating one of them will mutate the others in the dictionary. To resolve this issue, make deep copies using .copy(). You also should be using f-strings rather than .format(): for i in range(3): dict_strat[f'Df {i}'] = df.copy()
Question: I'm trying to minimize a function in order to better understand the optimizer process. As an example I used the Eggholder-Function (https://www.sfu.ca/~ssurjano/egg.html) which is 2d. My goal is to get the values of my parameters (x and y) after every optimizer iteration so that i can visualize it afterwards. Using Pytorch I wrote the following code: def eggholder_function(x): return -(x[1] + 47) * torch.sin(torch.sqrt(torch.abs(x[1] + x[0]/2 + 47))) - x[0]*torch.sin(torch.sqrt(torch.abs(x[0]-(x[1]+47)))) def minimize(function, initial_parameters): list_params = [] params = initial_parameters params.requires_grad_() optimizer = torch.optim.Adam([params], lr=0.1) for i in range(5): optimizer.zero_grad() loss = function(params) loss.backward() optimizer.step() list_params.append(params) return params, list_params starting_point = torch.tensor([-30.,-10.]) minimized_params, list_of_params = minimize(eggholder_function, starting_point)  The output is as follows: minimized_params: tensor([-29.4984, -10.5021], requires_grad=True) and list of params: [tensor([-29.4984, -10.5021], requires_grad=True), tensor([-29.4984, -10.5021], requires_grad=True), tensor([-29.4984, -10.5021], requires_grad=True), tensor([-29.4984, -10.5021], requires_grad=True), tensor([-29.4984, -10.5021], requires_grad=True)]  While I understand that the minimized_params is infact the optimized minimum, why does list_of_params show the same values for every iteration? Thank you and have a great day!

Answer: Because they all refer to the same object. You can check it by: id(list_of_params[0]), id(list_of_params[1])  You can clone the params to avoid that: import torch def eggholder_function(x): return -(x[1] + 47) * torch.sin(torch.sqrt(torch.abs(x[1] + x[0]/2 + 47))) - x[0]*torch.sin(torch.sqrt(torch.abs(x[0]-(x[1]+47)))) def minimize(function, initial_parameters): list_params = [] params = initial_parameters params.requires_grad_() optimizer = torch.optim.Adam([params], lr=0.1) for i in range(5): optimizer.zero_grad() loss = function(params) loss.backward() optimizer.step() list_params.append(params.detach().clone()) #here return params, list_params starting_point = torch.tensor([-30.,-10.]) minimized_params, list_of_params = minimize(eggholder_function, starting_point)  #list_params [tensor([-29.9000, -10.1000]), tensor([-29.7999, -10.2001]), tensor([-29.6996, -10.3005]), tensor([-29.5992, -10.4011]), tensor([-29.4984, -10.5021])]
Question: I have a large array of integers (~3k items), and I want to find the indices of every combination of numbers where the sum of said numbers is equal to X. How to do this without the program taking years to execute? I can find the first combination possible with the following Python code: def find_numbers_that_sum_to(source_arr, target_number, return_index=True): result = [] result_indices = [] for idx, item in enumerate(source_arr): sum_list = sum(result) assert (sum_list + item) <= target_number result.append(item) result_indices.append(idx) if (sum_list + item) == target_number: break return result_indices  But I need every combination possible. At least a way of generating them dynamically. I only ever need one of them at a time, but if the indices it gave me don't match another criteria I need, I'll need the next set.

Answer: Unfortunately, this problem is NP-hard, meaning that there's no currently known polynomial time algorithm for this. If you want to benchmark your current code against other implementations, this SO question contains a bunch. Those implementations may be faster, but their runtimes will still be exponential.
Question: I'm using poetry to manage my python project, here's the project: my_project/ ├── pyproject.toml ├── module.py └── scripts/ └── main.py  And I want to know how to import function from module.py into my_scripts/main.py correctly. My pyproject.toml: [tool.poetry] name = "my_project" version = "0.1.0" description = "" authors = [] [tool.poetry.dependencies] python = "^3.11" [build-system] requires = ["poetry-core"] build-backend = "poetry.core.masonry.api"  I have tried this: # In my_scripts/main.py from module import my_function  And run these commands: poetry install poetry shell python my_scripts/main.py  then got this error: ModuleNotFoundError: No module named 'module'  I also have put a __init__.py under my_project/ but didn't work out.

Answer: What made it work for me is to create a folder in your root with the same name of the package in the pyproject.toml file. If you don't do this, poetry will not find your project and will not install it in editable mode. You also need to let Python know that a folder is a package or sub-package by placing an __init__.py file in it. my_project/ ├─ pyproject.toml ├─ my_project/ ├─ __init__.py ├─ module.py ├─ scripts/ ├─ __init__.py ├─ main.py  When you initialize the folder you should see that the project is installed. ...> poetry install Installing dependencies from lock file Installing the current project: my_project (0.1.0)  Last thing, you need to change the import in main.py. # main.py from my_project.module import my_function print("my_function imported")  You can now activate the virtual environment (if it's not already active) and run your script. ...> poetry shell Spawning shell within: C:\...\my_project\.venv ... ...> python my_project\scripts\main.py my_function imported  PS As @sinoroc pointed out in the comments, it is possible (and preferable according to him) to run the module with python -m my_project.scripts.main. I am new to the topic, but according to this SO answer:  __package__ is set to the immediate parent package in <modulename> [when running python via command line with the -m option]  This means that relative imports like the following magically work. # main.py from ..module import my_function # ..module tells python to look in the parent directory for module.py print("my_function imported")  ...> python -m my_project.scripts.main my_function imported ...> python my_project\scripts\main.py Traceback (most recent call last): ... ImportError: attempted relative import with no known parent package
Question: Python's Itertools has what is called starmap. Given a collection of collections and a function, it applies the function to each collection strictly inside the collection, using the elements of said internal collection as arguments to the function. For example, from itertools import starmap NestedList = [(1, 2), (3, 4), (5, 6), (0, 0), (1, 1), (2, 2)] list(starmap(lambda x, y:x + y, NestedList))  returns the list containing 3, 7, 11, 0, 2, and 4. I refuse to believe that Python was the first to come up with this concept, but I'm drawing a blank when I try to think of what it was called in older languages. Does any analogous functionality exist in common lisp? I feel certain that it does, but I cannot name it.

Answer: Use a combination of mapcar and apply: (defun starmap (f list) (mapcar (lambda (x) (apply f x)) list))  Or loop with the keyword collect: (defun starmap (f list) (loop for x in list collect (apply f x)))  Examples: > (starmap (lambda (x y) (+ x y)) '((1 2) (3 4) (5 6) (0 0) (1 1) (2 2))) (3 7 11 0 2 4) > (starmap #'expt '((1 2) (3 4) (5 6) (0 0) (1 1) (2 2))) (1 81 15625 1 1 4)
Question: I have installed opencv-python-4.7.0.68 and opencv-contrib-python-4.7.0.68 The code below gives me the following error: AttributeError: module 'cv2.aruco' has no attribute 'CharucoBoard_create' Sample code: import cv2 aruco_dict = cv2.aruco.getPredefinedDictionary(cv2.aruco.DICT_4X4_50) board = cv2.aruco.CharucoBoard_create(11, 8, 0.015, 0.011, aruco_dict) 

Answer: This is due to a change that happened in release 4.7.0, when the Aruco code was moved from contrib to the main repository. The constructor cv2.aruco.CharucoBoard_create has been renamed to cv2.aruco.CharucoBoard and its parameter list has changed slightly -- instead of the first two integer parameters squaresX and squaresY, you should pass in a single tuple with two values, representing the size. (Note: The documentation seems to be missing the Python constructor's signature. Bug report has been filed.) So, your code should look like: import cv2 aruco_dict = cv2.aruco.getPredefinedDictionary(cv2.aruco.DICT_4X4_50) board = cv2.aruco.CharucoBoard((11, 8), 0.015, 0.011, aruco_dict)
Question: I tried to install bipwallet through pip but it says there is no 'build_py_2to3' in distutils Defaulting to user installation because normal site-packages is not writeable Collecting bipwallet ... Collecting protobuf==3.0.0a3 Using cached protobuf-3.0.0a3.tar.gz (88 kB) Preparing metadata (setup.py) ... error error: subprocess-exited-with-error × python setup.py egg_info did not run successfully. │ exit code: 1 ╰─> [6 lines of output] Traceback (most recent call last): File "<string>", line 2, in <module> File "<pip-setuptools-caller>", line 34, in <module> File "/tmp/pip-install-q8v8yny3/protobuf_3f1a8b67130540ab9c93af7fe765918c/setup.py", line 29, in <module> from distutils.command.build_py import build_py_2to3 as _build_py ImportError: cannot import name 'build_py_2to3' from 'distutils.command.build_py' (/home/orkhan/.local/lib/python3.11/site-packages/setuptools/_distutils/command/build_py.py) [end of output] note: This error originates from a subprocess, and is likely not a problem with pip. error: metadata-generation-failed × Encountered error while generating package metadata. ╰─> See above for output. note: This is an issue with the package mentioned above, not pip. hint: See above for details.  I tried to search in Google but it did not help. I also tried pip install --upgrade distutils thinking maybe it's just older version. P.S. my python version 3.11

Answer: It seems as though bipwallet or one of its dependencies (protobuf-3.0.0a3?) wants to use whatever version of setuptools is available rather than pinning a specific version. setuptools v58.0.0 has a breaking change, first included in Python 3.10, where build_py_2to3 was removed. You have a couple options:  Find the offending library and edit its setup.py to indicate that it should use setuptools<=57.5.0 and retry. Downgrade your Python installation to 3.9 to get a local version of setuptools prior to the breaking change.  Here are a couple other related posts/links to the issue you're seeing:  https://github.com/mhammond/pywin32/issues/1813 https://bytemeta.vip/repo/StanfordVL/iGibson/issues/227 https://github.com/mobinmbn/bipwallet_fix
Question: I have the following dataframe:      Audi Hyundai Kia Mercedes Tesla VW Volvo     2019 0.25 nan nan 0.5 nan nan 0.25   2020 nan 0.125 nan 0.375 0.125 0.125 0.25   2021 nan nan 0.25 0.5 nan 0.25 nan     I want to rearrange the columns such the the first row is sorted from largest to smallest. So the order of the columns should be Mercedes, Audi/Volvo, the rest. I tried df.sort_values() so many times, but I always get errors. The most common error is about the usage of by.

Answer: You can reorder the columns based on the sorted order of the first row: out = df[df.iloc[0].sort_values(ascending=False).index] print(out) # Output Mercedes Audi Volvo Hyundai Kia Tesla VW 2019 0.500 0.25 0.25 NaN NaN NaN NaN 2020 0.375 NaN 0.25 0.125 NaN 0.125 0.125 2021 0.500 NaN NaN NaN 0.25 NaN 0.250
Question: Anyone please help me how to remove the lines link scatter points when plot with graph objects from Python enter image description here ` fig.add_trace(go.Scatter3d(x=x[65000:133083], y=y[65000:133083], z=z[65000:133083], marker=dict( size=1, # Changed node size... color=color[65000:133083], # ...and color colorscale='Viridis', line=None, showscale=True)))`  Please help me solve this

Answer: Have you tried to add: mode = 'markers', marker = dict( size = 12, # color = z, # set color to an array/list of desired values colorscale = "Viridis", # choose a colorscale opacity = 0.8 )  Though i'm not sure why you do have lines given your code...
Question: I'm trying to convert string timestamps to polars datetime from the timestamps my camera puts in it RAW file metadata, but polars throws this error when I have timestamps from both summer time and winter time. ComputeError: Different timezones found during 'strptime' operation.  How do I persuade it to convert these successfully? (ideally handling different timezones as well as the change from summer to winter time) And then how do I convert these timestamps back to the proper local clocktime for display? Note that while the timestamp strings just show the offset, there is an exif field "Time Zone City" in the metadata as well as fields with just the local (naive) timestamp import polars as plr testdata=[ {'name': 'BST 11:06', 'ts': '2022:06:27 11:06:12.16+01:00'}, {'name': 'GMT 7:06', 'ts': '2022:12:27 12:06:12.16+00:00'}, ] pdf = plr.DataFrame(testdata) pdfts = pdf.with_column(plr.col('ts').str.strptime(plr.Datetime, fmt = "%Y:%m:%d %H:%M:%S.%f%z")) print(pdf) print(pdfts)  It looks like I need to use tz_convert, but I cannot see how to add it to the conversion expression and what looks like the relevant docpage just 404's broken link to dt_namespace

Answer: polars 0.16 update Since PR 6496, was merged you can parse mixed offsets to UTC, then set the time zone: import polars as pl pdf = pl.DataFrame([ {'name': 'BST 11:06', 'ts': '2022:06:27 11:06:12.16+01:00'}, {'name': 'GMT 7:06', 'ts': '2022:12:27 12:06:12.16+00:00'}, ]) pdfts = pdf.with_columns( pl.col('ts').str.to_datetime("%Y:%m:%d %H:%M:%S%.f%z") .dt.convert_time_zone("Europe/London") ) print(pdfts) shape: (2, 2) ┌───────────┬─────────────────────────────┐ │ name ┆ ts │ │ --- ┆ --- │ │ str ┆ datetime[μs, Europe/London] │ ╞═══════════╪═════════════════════════════╡ │ BST 11:06 ┆ 2022-06-27 11:06:12.160 BST │ │ GMT 7:06 ┆ 2022-12-27 12:06:12.160 GMT │ └───────────┴─────────────────────────────┘   old version: Here's a work-around you could use: remove the UTC offset and localize to a pre-defined time zone. Note: the result will only be correct if UTC offsets and time zone agree. timezone = "Europe/London" pdfts = pdf.with_column( plr.col('ts') .str.replace("[+|-][0-9]{2}:[0-9]{2}", "") .str.strptime(plr.Datetime, fmt="%Y:%m:%d %H:%M:%S%.f") .dt.tz_localize(timezone) ) print(pdf) ┌───────────┬──────────────────────────────┐ │ name ┆ ts │ │ --- ┆ --- │ │ str ┆ str │ ╞═══════════╪══════════════════════════════╡ │ BST 11:06 ┆ 2022:06:27 11:06:12.16+01:00 │ │ GMT 7:06 ┆ 2022:12:27 12:06:12.16+00:00 │ └───────────┴──────────────────────────────┘ print(pdfts) ┌───────────┬─────────────────────────────┐ │ name ┆ ts │ │ --- ┆ --- │ │ str ┆ datetime[ns, Europe/London] │ ╞═══════════╪═════════════════════════════╡ │ BST 11:06 ┆ 2022-06-27 11:06:12.160 BST │ │ GMT 7:06 ┆ 2022-12-27 12:06:12.160 GMT │ └───────────┴─────────────────────────────┘  Side-Note: to be fair, pandas does not handle mixed UTC offsets either, unless you parse to UTC straight away (keyword utc=True in pd.to_datetime). With mixed UTC offsets, it falls back to using series of native Python datetime objects. That makes a lot of the pandas time series functionality like the dt accessor unavailable.
Question: I have a .env defined with the following content: env=loc  I have three python module that make use of this variable. ├── __init__.py ├── cli.py |── settings.py ├── commands │ ├── __init__.py │ └── output.py  settings.py: from dotenv import load_dotenv load_dotenv() if not os.getenv("env"): raise TypeError("'env' variable not found in .env file")  output.py: import os def output(): return getenv("env")  cli.py: import settings from commands.output import output import os CURR_ENV = getenv("env") print(CURR_ENV) print(output())  Output: loc None  Why is the output from output.py not loc? The environment variables were loaded when load_dotenv() was run for the first time. Do I have to run load_dotenv() every time I need to access the environment variables?

Answer: This answer more or less repeats what's mentioned in the comments and adds a demo example. Once load_dotenv() is called, environment variables will be visible in the process it's called in (and in any child process) from that point. Suppose you have a project organized as follows. ├── commands │ ├── __init__.py │ └── output.py ├── __init__.py ├── .env ├── main.py |── settings.py  settings.py: from os import getenv print("From settings.py:", getenv("env"))  output.py: from os import getenv def output(): print("From output.py:", getenv("env")) output()  main.py: from os import getenv from dotenv import load_dotenv import settings # <-- settings don't see .env yet from commands.output import output # <-- output don't see .env yet load_dotenv() # <-- because load_dotenv is called here print("From main.py:", getenv("env")) output() # <--- the function call is made after load_dotenv(), so .env is visible  So if you run main.py in the CLI, you'll get the following output. PS path\to\module> python .\main.py From settings.py: None From output.py: None From main.py: loc From output.py: loc  To make .env visible everywhere, move load_dotenv logic before any of the imports in main.py or make a call to load_dotenv inside the first import (in this example, settings.py) instead of main.py because every line in settings.py will be read once it's imported so every import and function call in it will be read in main.py anyway.  In the above example of executing main.py, .env became visible when calling the output() function inside main.py because it was called after the call to load_dotenv. That means, if you want to execute output.py by itself, you'll need to import load_dotenv and make a call to it inside output.py in order for it to see the environment variables. This becomes somewhat useful if you have a test/ directory in your module and want to execute some file from there.
Question: The enum package in python 3.11 has the StrEnum class. I consider it very convenient but cannot use it in python 3.10. What would be the easiest method to use this class anyway?

Answer: On Python 3.10, you can inherit from str and Enum to have a StrEnum: from enum import Enum class MyEnum(str, Enum): choice1 = "choice1" choice2 = "choice2"  With this approach, you have string comparison: "choice1" == MyEnum.choice1 >> True  Be aware, however, that Python 3.11 makes a breaking change to classes which inherit from (str, Enum): https://github.com/python/cpython/issues/100458. You will have to change all instances of (str, Enum) to StrEnum when upgrading to maintain the same behavior. Or: you can execute pip install StrEnum and have this: from strenum import StrEnum class MyEnum(StrEnum): choice1 = "choice1" choice2 = "choice2"
Question: I know that i can do this : byte = 58 format ( byte , '08b' ) >>> '00111010'  with two bytes i have to do format( bytes , '016b') but if i doesn't have the number of bytes i can't set a number for format so i have to do : with open('file','rb')as a: b = a.read() c = int.from_bytes ( b ) d = format( c ,'b') d = (8-len(a)%8)*'0'+d  but i was wondering if there was easier way to do this and i want this without using any loops thanks!

Answer: You can map each byte to an 8-bit representation with the str.format method and then join the byte representations into a single string for output (where b is the bytes object you read from a file): print(''.join(map('{:08b}'.format, b)))