eshangj/stackoverflow_q_and_a_sample huggingface dataset:
Question: I have a python app with downstream dependencies on dynamic libraries that are available as X86 only. The app runs on a X86 MacOS machine, but on a ARM MacOS machine it fails with an ImportError. I've run lipo -archs on the libraries and they are x86_64 only. I have Python running in a virtualenv and it is a universal binary x86_64 arm64. The intermediary object file built by the application when it installs is also a universal binary x86_64 arm64. I suspect that Python is being run native as an ARM app, but because of the dependencies I need it run as an X86 app. Is there a MacOS or Rosetta2 option or environmental setting that I can use that would force the X86 Python binary to be executed as opposed to the ARM binary?

Answer: Looks like the only way to do this is to install a X86 version of python. I found a how to guide here - https://towardsdatascience.com/how-to-use-manage-multiple-python-versions-on-an-apple-silicon-m1-mac-d69ee6ed0250 but couldn't quite get the pyenv build part to work. So in the Rosetta i386 terminal I brew86 installed python. This put a X86 version of python into /usr/local/bin/python3 from which I was able to create a X86 only virtualenv. Broadly the steps are from the above link (minus the pyenv parts):  Install Rosetta Create a Rosetta terminal Install X86 homebrew in the Rosetta terminal Create an alias for the X86 homebrew in /usr/local/bin/brew Use the X86 brew to install X86 python (ends up /usr/local/bin/python3) Create a virtualenv based on the X86 python path pip install
Question: When I try to run any python code in debug mode using VScode, I got an error message saying: 42737 -- /home/<username>/Desktop/development/bopi/experiment_handler.py .vscode-server/extensions/ms-python.python-2022.4.0/pythonFiles/lib/python/debugpy/launcher 4 Traceback (most recent call last): File "/usr/lib/python2.7/runpy.py", line 174, in _run_module_as_main "__main__", fname, loader, pkg_name) File "/usr/lib/python2.7/runpy.py", line 72, in _run_code exec code in run_globals File "/home/<username>/.vscode-server/extensions/ms-python.python-2022.4.0/pythonFiles/lib/python/debugpy/__main__.py", line 43, in <module> from debugpy.server import cli File "/home/<username>/.vscode-server/extensions/ms-python.python-2022.4.0/pythonFiles/lib/python/debugpy/../debugpy/server/__init__.py", line 9, in <module> import debugpy._vendored.force_pydevd # noqa File "/home/<username>/.vscode-server/extensions/ms-python.python-2022.4.0/pythonFiles/lib/python/debugpy/../debugpy/_vendored/force_pydevd.py", line 37, in <module> pydevd_constants = import_module('_pydevd_bundle.pydevd_constants') File "/usr/lib/python2.7/importlib/__init__.py", line 37, in import_module __import__(name) File "/home/<username>/.vscode-server/extensions/ms-python.python-2022.4.0/pythonFiles/lib/python/debugpy/_vendored/pydevd/_pydevd_bundle/pydevd_constants.py", line 362, in <module> from _pydev_bundle._pydev_saved_modules import thread, threading File "/home/<username>/.vscode-server/extensions/ms-python.python-2022.4.0/pythonFiles/lib/python/debugpy/_vendored/pydevd/_pydev_bundle/_pydev_saved_modules.py", line 97, in <module> import queue as _queue; verify_shadowed.check(_queue, ['Queue', 'LifoQueue', 'Empty', 'Full', 'deque']) File "/home/<username>/.vscode-server/extensions/ms-python.python-2022.4.0/pythonFiles/lib/python/debugpy/_vendored/pydevd/_pydev_bundle/_pydev_saved_modules.py", line 75, in check raise DebuggerInitializationError(msg) _pydev_bundle._pydev_saved_modules.DebuggerInitializationError: It was not possible to initialize the debugger due to a module name conflict. i.e.: the module "queue" could not be imported because it is shadowed by: /home/<username>/.local/lib/python2.7/site-packages/queue/__init__.pyc Please rename this file/folder so that the original module from the standard library can be imported.  Deleting the init.pyc and init.py resulting with an error message about missing queue import.

Answer: Downgrading my Python extension in Visual Studio Code to v2022.2.1924087327 worked for me. Elevating @Onur Berk's comment below as part of the answer:  Its is very easy to downgrade the python extension, just click 'extensions' and find the Python extension and select it. Rather than clicking 'uninstall' click the arrow next to it, this will give you an option to install another version
Question: I am trying to host a webscraping function on aws lambda and am running into webdriver errors for selenium. Could someone show me how you go about adding the chromedriver.exe file and how do you get the pathing to work in AWS Lambda function. This is the portion of my function that has to do with selenium, from selenium import webdriver from selenium.webdriver.common.by import By from webdriver_manager.chrome import ChromeDriverManager from selenium.webdriver.support.ui import WebDriverWait from selenium.webdriver.support import expected_conditions as EC from selenium.webdriver.support.ui import Select from selenium.webdriver.chrome.service import Service import pandas as pd import mysql.connector from sqlalchemy import create_engine url = 'https://covid19criticalcare.com/pharmacies/' driver = webdriver.Chrome(service=Service(ChromeDriverManager().install())) driver.maximize_window() driver.get(url) wait = WebDriverWait(driver, 5)   I tried creating a lambda layer with the chromedriver.exe file  I followed this guide (https://dev.to/awscommunity-asean/creating-an-api-that-runs-selenium-via-aws-lambda-3ck3) but I couldn't add the headless chromium because of the file size pushing me over my function limit (my pandas and numpy dependence layers have taken up most of my space)  I tried driver = webdriver.Chrome(with a path variable) and tried different pathing but wasn't sure what the beginning of the path would be since its on a lambda function.  

Answer: I've been struggling adding selenium to the aws lambda for last couple days. I have a web scraping function (uses selenium and google api) which extracts data from a website and writes the outputs to a google spreadsheet. Let me explain what i did step by step and how i finally succeeded so you don't have to deal with it as much as me: 1- I tried to add selenium as a layer described here https://www.youtube.com/watch?v=jWqbYiHudt8. What i ended up was, i was succesfull with adding selenium but deployment package is over 250mb (describe lambda quotaas here: How to increase the maximum size of the AWS lambda deployment package (RequestEntityTooLargeException)?) so it did not work. 2- To overcome deployment package size, it is a good option to add as container images(10 gb deployment package size limit). Here is a good explanation of adding as container images https://cloudbytes.dev/snippets/run-selenium-in-aws-lambda-for-ui-testing#using-the-github-repository-directly . i tried it but i could not able to deploy as described due to missing/wrong webdrivers(the shell script seems to be wrong) 3- And finally, i was fully able to publish my selenium function as docker image as described here https://github.com/umihico/docker-selenium-lambda. There are lots of discussions about which version work with what. The most important issue about selenium is, you have to be careful about package and driver version when deploying to aws lambda.
Question: Given an array I should compute the following sum in linear time:  My most naive implementation is O(n3): sum_ = 0 for i in range(n): for j in range(n, i, -1): sum_ += max(arr[i:j]) * (j-i)  I have no idea what to do. I have tried many algorithms but they were at best O(n*log(n)), but I should solve it in linear time. Also, I don't get the idea, is there a mathematical way of just looking at an array and telling the result of the above sum?

Answer: Keep a stack of (indices of) non-increasing values. So before appending the new value, pop smaller ones. Whenever you pop one, add its contribution to the total. def solution(arr): arr.append(float('inf')) I = [-1] total = 0 for i in range(len(arr)): while arr[i] > arr[I[-1]]: j = I.pop() a = j - I[-1] b = i - j total += (a+b)*a*b//2 * arr[j] I.append(i) arr.pop() return total   The bars represent values, larger values are larger bars. The value at i is about to be added. The light grey ones come later. The green ones are on the stack. The brown ones already don't play a role anymore. First the one at i-1 gets popped, but that's less informative. Then the one at j gets popped. It dominates the range between I[-1] and i: it's the maximum in all subarrays in that range that contain it. These subarrays contain j as well as 0 to a-1 more elements to the left and 0 to b-1 more elements to the right. That's a*b subarrays and their average length is (a+b)/2. I temporarily append infinity to the values so it works as a sentinel on the left (avoids an extra check in the while condition) and as a cleaner at the end (it causes all remaining values to get popped from the stack). Non-Python-coders: Python supports negative indexes, -1 means "last element" (1st from the end). Correctness test with random lists of 500 values (Try it online!): import random def reference(arr): n = len(arr) return sum(max(arr[L : R+1]) * (R - (L-1)) for L in range(n) for R in range(L, n)) for _ in range(5): arr = random.choices(range(10000), k=500) expect = reference(arr) result = solution(arr) print(result == expect, result)  Sample output (results for five lists, True means it's correct): True 207276773131 True 208127393653 True 208653950227 True 208073567605 True 206924015682
Question: I am getting an error on my modeling of lightgbm searching for optimal auc. Any help would be appreciated. import optuna from sklearn.model_selection import StratifiedKFold from optuna.integration import LightGBMPruningCallback def objective(trial, X, y): param = { "objective": "binary", "metric": "auc", "verbosity": -1, "boosting_type": "gbdt", "lambda_l1": trial.suggest_loguniform("lambda_l1", 1e-8, 10.0), "lambda_l2": trial.suggest_loguniform("lambda_l2", 1e-8, 10.0), "num_leaves": trial.suggest_int("num_leaves", 2, 256), "feature_fraction": trial.suggest_uniform("feature_fraction", 0.4, 1.0), "bagging_fraction": trial.suggest_uniform("bagging_fraction", 0.4, 1.0), "bagging_freq": trial.suggest_int("bagging_freq", 1, 7), "min_child_samples": trial.suggest_int("min_child_samples", 5, 100), } cv = StratifiedKFold(n_splits=5, shuffle=True, random_state=1121218) cv_scores = np.empty(5) for idx, (train_idx, test_idx) in enumerate(cv.split(X, y)): X_train, X_test = X.iloc[train_idx], X.iloc[test_idx] y_train, y_test = y[train_idx], y[test_idx] pruning_callback = optuna.integration.LightGBMPruningCallback(trial, "auc") model = lgb.LGBMClassifier(**param) model.fit( X_train, y_train, eval_set=[(X_test, y_test)], early_stopping_rounds=100, callbacks=[pruning_callback]) preds = model.predict_proba(X_test) cv_scores[idx] = log_loss(y_test, preds) auc_scores[idx] = roc_auc_score(y_test, preds) return np.mean(cv_scores), np.mean(auc_scores) study = optuna.create_study(direction="minimize", study_name="LGBM Classifier") func = lambda trial: objective(trial, sample_df[cols_to_keep], sample_df[target]) study.optimize(func, n_trials=1)   Trial 0 failed because of the following error: ValueError('The intermediate values are inconsistent with the objective values in terms of study directions. Please specify a metric to be minimized for LightGBMPruningCallback.',)* 

Answer: Your objective function returns two values but you specify only one direction when creating the study. Try this: study = optuna.create_study(directions=["minimize", "maximize"], study_name="LGBM Classifier")
Question: I'm currently self-studying functional programming by writing a monad library in python. And I'm having trouble with type hinting. So for example, there is a function filterM in Haskell with signature filterM :: (a -> m Bool) -> [a] -> m [a]  Ideally, if python can pattern match "subtypes" of a TypeVar by putting a bracket after it, then I should be able to do it with something like this: T = TypeVar('T') M = TypeVar('M', bound=Monad) def filterM(filter_func: Callable[[T], M[bool]], iterable: list[T]) -> M[list[T]]  But it seems that the above syntax wouldn't work. In fact, it seems like there is no way to "extract" the type of monad I pass in at all. Say I pass in a Callable[[int], Maybe[bool]], the best I achieved was to take the whole Maybe[bool] as a single TypeVar. Then there is no way to convert it to the correct output type Maybe[list[int]].

Answer: Currently, what you want cannot be done. You'll have to make a plan that doesn't require it.
Question: I have a conda environment that has a package pinned as follows: Pinned packages: - python 3.8.* - bcbio-gff 0.6.7.* - snakemake 6.7.0.*  How do I remove the pin for one of the pinned packages, just using command line conda / mamba? I've tried conda update snakemake but that doesn't remove the pin. I can change the pin easily, e.g. by conda install snakemake=7, but then I have snakemake still pinned. I want to unpin snakemake entirely. I had a look at potentially similar questions, but none seemed to answer my question.

Answer: This is only a suboptimal answer, but it's the best I could find so far: You need to manually remove the pinned package from a config file called pinned which you can find in CONDA_PATH/base/envs/ENV_NAME/conda-meta/pinned  In my case I had to do: vim /usr/local/Caskroom/mambaforge/base/envs/nextstrain/conda-meta/pinned  And remove the line: snakemake=6.7.0  It would be much nicer if there was a conda CLI command - but it doesn't seem to exist.
Question: I have a .pth file created with Pytorch with weights. How would I be able to view the weights from this file? I tried this code to load and view but it was not working (as a newbie, I might be entirely wrong)- import torch import torchvision.models as models torch.save('weights\kharif_crops_final.pth') models.load_state_dict(torch.load('weights\kharif_crops_final.pth')) models.eval() print(models) 

Answer: import torch model = torch.load('path') print(model)  (Verify and confirm)
Question: I am trying to use cache in my .gitlab-ci.yml file, but the time only increases (testing by adding blank lines). I want to cache python packages I install with pip. Here is the stage where I install and use these packages (other stages uses Docker): image: python:3.8-slim-buster variables: PIP_CACHE_DIR: "$CI_PROJECT_DIR/.cache/pip" cache: paths: - .cache/pip stages: - lint - test - build - deploy test-job: stage: test before_script: - apt-get update - apt-get install -y --no-install-recommends gcc - apt install -y default-libmysqlclient-dev - pip3 install -r requirements.txt script: - pytest tests/test.py  After running this pipeline, with each pipeline, the pipeline time just increases. I was following these steps from GitLab documentation - https://docs.gitlab.com/ee/ci/caching/#cache-python-dependencies Although I am not using venv since it works without it. I am still not sure why the PIP_CACHE_DIR variable is needed if it is not used, but I followed the documentation. What is the correct way to cache python dependencies? I would prefer not to use venv.

Answer: PIP_CACHE_DIR is a pip feature that can be used to set the cache dir. The second answer to this question explains it. There may be some disagreement on this, but I think that for something like pip packages or node modules, it is quicker to download them fresh for each pipeline. When the packages are cached by Gitlab by using cache: paths: - .cache/pip  The cache that Gitlab creates gets zipped and stored somewhere(where it gets stored depends on runner config). This requires zipping and uploading the cache. Then when another pipeline gets created, the cache needs to be downloaded and unpacked. If using a cache is slowing down job execution, then it might make sense to just remove the cache.
Question: I am running a simple CNN using Pytorch for some audio classification on my Raspberry Pi 4 on Python 3.9.2 (64-bit). For the audio manipulation needed I am using librosa. librosa depends on the numba package which is only compatible with numpy version <= 1.20. When running my code, the line spect_tensor = torch.from_numpy(spect).double()  throws the RuntimeError: RuntimeError: Numpy is not available  Searching the internet for solutions I found upgrading Numpy to the latest version to resolve that specific error, but throwing another error, because Numba only works with Numpy <= 1.20. Is there a solution to this problem which does not include searching for an alternative to using librosa?

Answer: Just wanted to give an update on my situation. I downgraded torch to version 0.9.1 which solved the original issue. Now OpenBLAS is throwing a warning because of an open MPLoop. But for now my code is up and running.
Question: I've done alot of searching to try and find a way to achieve this but the solutions I've found either don't do what I need or I don't understand them. I'm looking for a way of playing a sound in python (non-blocking) that allows me to change the playback speed in real time, as it's playing, with no gaps or cutouts. Changing the pitch is fine. Audio quality isn't even that important. Most of the solutions I've found only allow setting the playback speed once, before the file is played.

Answer: I've found a solution, using python-mpv, a wrapper for mpv.io from pynput.keyboard import Key, Listener import mpv speed=1 #quick function to change speed via keyboard. def on_press(key): global speed if key.char == 'f' : speed=speed-0.1 player.speed=speed if key.char == 'g' : speed=speed+0.1 player.speed=speed player = mpv.MPV(ytdl=True) player.play('/Users/regvardy/mediapipe_faceswap-main/test.wav') with Listener( on_press=on_press) as listener: listener.join() while True: player.speed=speed  I haven't tested it for stability yet. It feels like a workaround rather than me actually finding out how to do it so I may try and find a different solution.
Question: I have a mask active that tracks batches that still have not terminated in a recurrent process. It's dimension is [batch_full,], and it's true entries show which elements need to still be used in current step. The recurrent process generates another mask, terminated, which has as many elements as true values in active mask. Now, I want to take values from ~terminated and put them back into active, but at the right indices. Basically I want to do: import torch active = torch.ones([4,], dtype=torch.bool) active[:2] = torch.tensor(False) terminated = torch.tensor([True, False]) active[active] = ~terminated print(active) # expected [F, F, F, T]  However, I get error:  RuntimeError: unsupported operation: some elements of the input tensor and the written-to tensor refer to a single memory location. Please clone() the tensor before performing the operation.  How can I do the described above operation in an effective way?

Answer: There are a few solutions, I will also give their speed as measured by timeit, 10k repetitions, on 2021 macbook pro. The simplest solution, taking 0.260s: active[active.clone()] = ~terminated  We can use masked_scatter_ inplace operation for abt. 2x speedup (0.136s): active.masked_scatter_( active, ~terminated, )  Out of place operation, taking 0.161s, would be: active = torch.masked_scatter( active, active, ~terminated, )
Question: Let's say I have a list: list = [[1, 2, 3, 4], ['a', 'b', 'c', 'd'], [9, 8, 7, 6]]  and I would like to get something like: newList = [[2, 3, 4], ['b', 'c', 'd'], [8, 7, 6]]  hence I tried going with this solution print(list[0:][1:])  But I get this output [['a', 'b', 'c', 'd'], [9, 8, 7, 6]]  Therefore I tried print(list[1:][0:])  but I get precisely the same result. I tried to make some research and experiments about this specific subject but without any result.

Answer: You want the 1 to end element of every row in your matrix. mylist = [[1, 2, 3, 4], ['a', 'b', 'c', 'd'], [9, 8, 7, 6]] new_list = [row[1:] for row in mylist]
Question: Suppose I have the regular cartesian coordinate system $(x,y)$ and I consider a rectangular mesh region $D$ (split into little squares). I want to see how the domain D would be mapped under a coordinate transform T:(x,y) -> (u(x,y) ,v(x,y) ) in Python? I'm looking for something like this:  See here. Could I be advised on how this could be done? I am a total beginner at python and programming.

Answer: If I understand you correctly, you want to be able to see a sort of a grid plot of a transformed cartesian space? In that case, maybe something like this, using Numpy and Matplotlib. (You could do the same with just Pillow to draw some lines, but this is more convenient...) EDIT: Following the discussion in the comments, I changed this a bit compared to the simpler original (see edit history), to make it easier to plot multiple transformations, as well as to color the lines to make it easier to follow how they're transformed. It's still pretty simple, though. (For fun, I also threw in a couple of extra transformations.) import math import numpy as np import matplotlib.pyplot as plt from matplotlib import cm colormap = cm.get_cmap("rainbow") def plot_grid( xmin: float, xmax: float, ymin: float, ymax: float, n_lines: int, line_points: int, map_func, ): """ Plot a transformation of a regular grid. :param xmin: Minimum x value :param xmax: Maximum x value :param ymin: Minimum y value :param ymax: Maximum y value :param n_lines: Number of lines per axis :param line_points: Number of points per line :param map_func: Function to map the grid points to new coordinates """ # List for gathering the lines into. lines = [] # Iterate over horizontal lines. for y in np.linspace(ymin, ymax, n_lines): lines.append([map_func(x, y) for x in np.linspace(xmin, xmax, line_points)]) # Iterate over vertical lines. for x in np.linspace(xmin, xmax, n_lines): lines.append([map_func(x, y) for y in np.linspace(ymin, ymax, line_points)]) # Plot all the lines. for i, line in enumerate(lines): p = i / (len(lines) - 1) # Normalize to 0-1. # Transpose the list of points for passing to plot. xs, ys = zip(*line) # Get the line color from the colormap. plt.plot(xs, ys, color=colormap(p)) # Define some mapping functions. def identity(x, y): return x, y def example(x, y): c = complex(x, y) ** 2 return (c.real, c.imag) def wobbly(x: float, y: float): return x + math.sin(y * 2) * 0.2, y + math.cos(x * 2) * 0.3 def vortex(x: float, y: float): dst = (x - 2) ** 2 + (y - 2) ** 2 ang = math.atan2(y - 2, x - 2) return math.cos(ang - dst * 0.1) * dst, math.sin(ang - dst * 0.1) * dst # Set up the plot surface... plt.figure(figsize=(8, 8)) plt.tight_layout() plt.subplot(2, 2, 1) plt.title("Identity") plot_grid(0, 4, 0, 4, 10, 10, identity) plt.subplot(2, 2, 2) plt.title("Example") plot_grid(0, 4, 0, 4, 10, 10, example) plt.subplot(2, 2, 3) plt.title("Wobbly") plot_grid(0, 4, 0, 4, 10, 40, wobbly) plt.subplot(2, 2, 4) plt.title("Vortex") plot_grid(0, 4, 0, 4, 10, 40, vortex) plt.savefig("so71735261-2.png") plt.show()  The result image is:
Question: II currently have a very large .csv with 2 million rows. I've read in the csv and only have 2 columns, number and timestamp (in unix). My goal is to grab the last and largest number for each day (eg. 1/1/2021, 1/2/2021, etc.)  I have converted unix to datetime and used df.groupby('timestamp').tail(1) but am still not able to return the last row per day. Am I using the groupby wrong?  import pandas as pd def main(): df = pd.read_csv('blocks.csv', usecols=['number', 'timestamp']) print(df.head()) df['timestamp'] = pd.to_datetime(df['timestamp'],unit='s') x = df.groupby('timestamp').tail(1) print(x) if __name__ == '__main__': main()  Desired Output:  number timestamp 11,509,218 2021-01-01 11,629,315 2021-01-02 11,782,116 2021-01-03 12,321,123 2021-01-04 ... 

Answer: The "problem" lies in the grouper, use .dt.date for correct grouping (assuming your data is already sorted): x = df.groupby(df['timestamp'].dt.date).tail(1) print(x)