eshangj/stackoverflow_q_and_a_sample huggingface dataset:
Question: I recently updated the Python version of my base conda environment from 3.8 to 3.9, using mamba update python=3.9, but I can no longer run IPython, because the sqlite3 package appears to be broken. python Python 3.9.15 | packaged by conda-forge | (main, Nov 22 2022, 08:55:37) [Clang 14.0.6 ] on darwin Type "help", "copyright", "credits" or "license" for more information. >>> import sqlite3 Traceback (most recent call last): File "<stdin>", line 1, in <module> File "/Users/rosborn/opt/miniconda3/lib/python3.9/sqlite3/__init__.py", line 57, in <module> from sqlite3.dbapi2 import * File "/Users/rosborn/opt/miniconda3/lib/python3.9/sqlite3/dbapi2.py", line 27, in <module> from _sqlite3 import * ImportError: dlopen(/Users/rosborn/opt/miniconda3/lib/python3.9/lib-dynload/_sqlite3.cpython-39-darwin.so, 0x0002): Symbol not found: (_sqlite3_enable_load_extension) Referenced from: '/Users/rosborn/opt/miniconda3/lib/python3.9/lib-dynload/_sqlite3.cpython-39-darwin.so' Expected in: '/usr/lib/libsqlite3.dylib'  Since I had another Python 3.9 environment that is still functional, I tried copying over the envs/py39/lib/sqlite3.36.0 and envs/py39/lib/python3.9/sqlite3 directories, as well as envs/py39/lib/python3.9/lib-dynload/_sqlite3.cpython-39-darwin.so because I assumed the sqlite3 libraries had been incorrectly compiled, but that doesn't fix the problem. On the Homebrew Github, there was a related issue, where someone suggested checking whether the missing symbol was there. It seems to be all present and correct. $ nm -gj /Users/rosborn/opt/miniconda3/lib/python3.9/lib-dynload/_sqlite3.cpython-39-darwin.so | grep enable_load_extension _sqlite3_enable_load_extension  I don't know how Homebrew installs sqlite3, but the remaining fixes seemed to require checking the system libsqlite, which I don't have administrative access to. In case it's relevant, I am on an Intel Mac, so it's not related to the M1 chip, as some related issues appear to be. Does the conda distribution attempt to link to the system libsqlite? If so, why does this problem not affect the py39 environment? Any tips will be welcome. If it were not the base environment, I would just delete the one with the problem and start again. I attempted a forced reinstall of sqlite3, but it appeared not to be installable as a separate package.

Answer: Following the suggestions by @merv, the solution to this problem was to force a reinstall of the libsqlite package. $ mamba install libsqlite --force-reinstall  After updating Python, it seems that sqlite3 was linked to the Mac system library, /usr/lib/libsqlite3.dylib, rather than one installed by conda-forge. According to discussions elsewhere, it is likely that Apple disables the missing _sqlite3_enable_load_extension extension for security reasons, leading to the observed error message. I don't know why the link error occurred in the first place, but fortunately, conda distributes libsqlite as a separate package, so the fix was simple to implement.
Question: This is a serious question. Please do not take it as a joke. This is a scatter plot from an infamous paper with the same name, Get me off Your F****** Mailing List by Mazières and Kohle (2005), published in a predatory journal. Some people may know it.  I am seriously interested in recreating the same scatter plot to test a new density-based clustering algorithm without the need of creating all the letters from scratch. Is there any way to make this process easier? (e.g. a dataset, or a package, or a smart way to recreate the plot)

Answer: Now that the grid package supports clipping paths, we can do: library(grid) library(ggplot2) tg <- textGrob("Get me off\nYour Fuck\ning Mailing\nList", x = 0.2, hjust = 0, gp = gpar(cex = 6, col = "grey", font = 2)) cg <- pointsGrob(x= runif(15000), y = runif(15000), pch = 3, gp = gpar(cex = 0.5)) rg <- rectGrob(width = unit(0.5, 'npc'), height = unit(0.1, 'npc'), gp = gpar(fill = 'red')) ggplot(data = NULL, aes(x = 100, y = 100)) + geom_point(col = 'white') + theme_classic() + theme(panel.border = element_rect(fill = 'white', linewidth = 1)) pushViewport(viewport(clip = tg)) grid.draw(cg)
Question: I need to find all red pixels in an image and create my own image with just the red pixels. So far I have been experimenting with matplotlib as I am very new to it. def find_red_pixels(map, upper_threshold=100, lower_threshold =50): """Finds all red pixels of the "map" image and outputs a binary image file "map-red-pixels.jpg" """ red_map = [] for row in map: for pixel in row: print(pixel)  I am brand new to image manipulating and have tried this, however I do not understand what the values [0.0.01] etc. mean which is outputted as pixels. Is there an easy way to do this?

Answer: The first thing to understand is the way the image is represented in an array when it is read in. Here I read in a 320x160 image of a rainbow:  >>> img = plt.imread('rainbow.png') >>> img.shape (160, 320, 4)  This shows the dimensions of the array - note the last element of the tuple is 4. These values represent the red pixel values, the green pixel values, the blue pixel values, and the transparency or alpha values. We want to find pixels which have a certain red value, so we're just interested in the first of these. Let's split these out so we can just work with them: >>> r = img[:,:,0]  This isolates the portion of the array relating to the redness of the pixel values, represented by a value between 0 and 1. We now want to isolate the locations of the pixels which have a certain redness value. We can use np.where for this, finding the elements in our new array r which have values of over 0.5 (this value can be altered to fit your requirements of upper_threshold and lower_threshold), and replacing these with a value of 1, and 0 otherwise; creating a binary image. >>> binary_img = np.where(r > 0.5, 1, 0)  Finally, we just need to save this image, using plt.imsave: >>> plt.imsave('rainbow_red.png', binary_img , cmap='gray')  Giving the following result:
Question: After Installing cartopy on google-colab, I was not able to import it: !pip install cartopy import cartopy ImportError: cannot import name lgeos 

Answer: The best way to install Cartopy in a Colab is by using the Conda environment. So we need to install the following: #1|Install Conda environment on Colab !pip install -q condacolab import condacolab condacolab.install()  Then, #2|Install cartopy !mamba install -q -c conda-forge cartopy  After that, #3|imoprt cartopy import cartopy
Question: I am trying to do a list comprehension of a tuples list based on another list of tuples with partial match. x = [((1,1),(1,1),(1,2)), ((2,1),(1,3),(2,9)), ((2,1),(2,3),(2,9))] y = [(2,1),(1,3)] [i for i in x for k in y if k in i]  e.g. in this x is a list of tuples & y is the desired list of tuples. If y is found in any of the items in the list of tuples in x, it should return that item. Result is: [((2, 1), (1, 3), (2, 9)), ((2, 1), (1, 3), (2, 9)), ((2, 1), (2, 3), (2, 9))] But I want only : [((2, 1), (1, 3), (2, 9))] I tried with single tuple & it gave the desired result. But not sure why it doesn't work when I pass a tuple list. x = [((1,1),(1,1),(1,2)), ((2,1),(1,3),(2,9)), ((2,1),(2,3),(2,9))] y = (2,1) [i for i in x if y in i]  Result: [((2, 1), (1, 3), (2, 9)), ((2, 1), (2, 3), (2, 9))]

Answer: You can use: x = [ ((1, 1), (1, 1), (1, 2)), ((2, 1), (1, 3), (2, 9)), ((2, 1), (2, 3), (2, 9)), ] y = [(2, 1), (1, 3)] print([t for t in x if not set(y).difference(t)])  output: [((2, 1), (1, 3), (2, 9))]  If you subtract tuples inside x from y with set operations, if all the sub tuples are present, you'll end up with an empty set, so you want that tuple. You could also write if set(y).issubset(t) instead of if not set(y).difference(t) part.(borrowed from @kellyBundy's answer)  What if I want any tpl of y to be true but not all.  print([t for t in x if set(y).intersection(t)])
Question: I am new to this, trying to publish a package to pypi.org using Poetry package. on my local the build is working, I am able to import the package test run it, it's all good. but when I try to publish it to pypi.org, I get below error - as per the article I was following Link, it was supposed to prompt me for my pypi account ID and password, but it doesn't and then gives the error:  Publishing gsst (0.2.2) to PyPI - Uploading gsst-0.2.2-py3-none-any.whl 0% - Uploading gsst-0.2.2-py3-none-any.whl 100%  and then this error shows up -- HTTP Error 403: Invalid or non-existent authentication information. See https://pypi.org/help/#invalid-auth for more information. | b'<html>\n <head>\n <title>403 Invalid or non-existent authentication information. See https://pypi.org/help/#invalid-auth for more information.\n \n <body>\n <h1>403 Invalid or non-existent authentication information. See https://pypi.org/help/#invalid-auth for more information.\n Access was denied to this resource.<br/><br/>\nInvalid or non-existent authentication information. See https://pypi.org/help/#invalid-auth for more information.\n\n\n \n'  after i run the -- poetry publish command, the CLI should prompt me for pypi, ID and password. why does it skip it and then fails on authentication.

Answer: I was able to resolve the problem finally - Took help from poetry documentation here Issued the below command to setup my pypi.org account for auto authentication poetry config http-basic.pypi <username> <password>  After that I ran the "poetry publish" command and was able to publish my package on pypi.org. It's really the quickest and easiest way to post your package to pypi.org
Question: When I use bitwise and operator(&) with the number of 1 to find out if a number x is odd or even (x & 1), does the interpreter change the binary representation of 1 according to the binary representation of x? For example:  2 & 1 -> 10 & 01 -> then perform comparison bitwise 5 & 1 -> 101 & 001 -> then perform comparison bitwise 100 & 1 -> 1100100 & 0000001 -> then perform comparison bitwise  Does it append zeros to the binary representation of 1 to perform bitwise and operation? Looking at the cpython implementation, it looks like that it compares the digits according to size of the right argument. So in this case the example above works actually:  2 & 1 -> 10 & 1 -> 0 & 1 -> then perform comparison bitwise 5 & 1 -> 101 & 1 -> 1 & 1 -> then perform comparison bitwise 100 & 1 -> 1100100 & 1 -> 0 & 1 -> then perform comparison bitwise  Is my understanding right? I'm confused because of this image from Geeks for Geeks.

Answer: Conceptually, adding zeros to the shorter number gives the same result as ignoring excess digits in the longer number. They both do the same thing. Padding, however, is inefficient, so in practice you wouldn't want to do it. The reason is because anything ANDed with 0 is 0. If you pad the short number to match the longer one and then AND the extra bits they're all going to result in 0. It works, but since you know the padded bits will just result in extra zeros, it's more efficient to ignore them and only iterate over the length of the shorter number. Python only processes the overlapping digits. First, it conditionally swaps a and b to ensure b is the smaller number: /* Swap a and b if necessary to ensure size_a >= size_b. */ if (size_a < size_b) { z = a; a = b; b = z; size_z = size_a; size_a = size_b; size_b = size_z; negz = nega; nega = negb; negb = negz; }  Then it iterates over the smaller size_b: /* Compute digits for overlap of a and b. */ switch(op) { case '&': for (i = 0; i < size_b; ++i) z->ob_digit[i] = a->ob_digit[i] & b->ob_digit[i]; break;   So my understanding is right, the image is just for intuition?  Yep, correct. The image is for conceptual understanding. It doesn't reflect how it's actually implemented in code.
Question: I am wondering what the fastest way for a mean computation is in numpy. I used the following code to experiment with it: import time n = 10000 p = np.array([1] * 1000000) t1 = time.time() for x in range(n): np.divide(np.sum(p), p.size) t2 = time.time() print(t2-t1)  3.9222593307495117 t3 = time.time() for x in range(n): np.mean(p) t4 = time.time() print(t4-t3)  5.271147012710571 I would assume that np.mean would be faster or at least equivalent in speed, however it looks like the combination of numpy functions is faster than np.mean. Why is the combination of numpy functions faster?

Answer: For integer input, by default, numpy.mean computes the sum in float64 dtype. This prevents overflow errors, but it requires a conversion for every element. Your code with numpy.sum only converts once, after the sum has been computed.
Question: I appreciate your help in advance. This is a practice question from Meta's interview preparation website. I have solved it, but I wonder if any optimization can be done. Question: Is there a way to solve the following problem with a time complexity of O(n)? Problem Description:  You have been given an array nums of type int. Write a program that returns the bool type as the return value of the solution function to determine whether it is possible to split nums into two arrays A and B such that the following two conditions are satisfied.  The sum of the respective array elements of A and B is equal. All the array elements in A are strictly smaller than all the array elements in B.   Examples:  nums = [1,5,7,1] -> true since A = [1,1,5], B = [7] nums = [12,7,6,7,6] -> false since A = [6,6,7], B = [7,12] failed the 2nd requirement  What I have tried: I have used the sort function in Python, which has a time complexity of O(nlog(n)). from typing import List def solution(nums: List[int]) -> bool: total_sum = sum(nums) # If the total sum of the array is 0 or an odd number, # it is impossible to have array A and array B equal. if total_sum % 2 or total_sum == 0: return False nums.sort() curr_sum, i = total_sum, 0 while curr_sum > total_sum // 2: curr_sum -= nums[i] i += 1 if curr_sum == total_sum // 2 and nums[i] != nums[i - 1]: return True return False 

Answer: For what it's worth, you can modify QuickSelect to get a with-high-probability and expected O(n)-time algorithm, though Python's sort is so fast that it hardly seems like a good idea. Deterministic O(n) is possible and left as an easy exercise to the reader familiar with selection algorithms (but the constant factor is even worse, so...). import random def can_partition(nums, a_sum=0, b_sum=0): if not nums: # True makes more sense here, but whatever return False pivot = random.choice(nums) less = sum(n for n in nums if n < pivot) equal = sum(n for n in nums if n == pivot) greater = sum(n for n in nums if n > pivot) a_ext = a_sum + less b_ext = greater + b_sum if abs(a_ext - b_ext) == equal: return True elif a_ext < b_ext: return can_partition([n for n in nums if n > pivot], a_ext + equal, b_sum) else: return can_partition([n for n in nums if n < pivot], a_sum, equal + b_ext) print(can_partition([1, 5, 7, 1])) print(can_partition([12, 7, 6, 7, 6]))
Question: I'm testing my stock trading logic and I made a position column to check the buying / selling signal df = pd.DataFrame({'position': [1.0, 0.0, 0.0, 1.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, -1.0, 1.0, 0.0, -1.0, -1.0, 0.0, 0.0, 0.0, 0.0, 0.0, 1.0]})  I want to replace 1.0 value occurs between 1.0 and -1.0 with 0.0, and replace -1.0 value occurs between -1.0 and 1.0 with 0.0 here is the desired output: df = pd.DataFrame({'position': [1.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, -1.0, 1.0, 0.0, -1.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 1.0]})  NOTE: the output only keeps the initial signal of 1.0 or -1.0

Answer: Here is a basic implementation based on the approach described by the previous answer: lastseen = 0 for n,el in enumerate(df["position"]): if lastseen == 0 and el == -1: raise Exception("Inconsistent data") if (el in [1, -1] and el != lastseen) or lastseen == 0: lastseen = el else: df["position"][n] = 0  I added the first check by considering the domain you described. If it's not correct for your problem feel free to remove it
Question: I have a Django application where all the secret information (secret key and keys for encryption) are in the .env file as environment variables - I'm using the python-dotenv library. After starting the application, I removed the .env file from the server files and the application continues to work as it should. Can deleting this file cause any problems? Is there any other (or better) way to secure these secrets in a Django application? If it's relevant, I use pythonanywhere.com

Answer: You shouldn't need .env file if you instead set up Environment variables while initialising the server machine. Many Cloud Service Providers let you do that. If you're setting up a docker container in Google App run, you should be able to setup environment variables or when setting up virtual machine with predetermined environment variables. This should eliminate any requirement for having .env file.
Question: I'm trying to use a session with autocommit=true to create a row in a table, and it does not seem to be working. The row is not saved to the table. import os import sqlalchemy from sqlalchemy import Table from sqlalchemy.ext.declarative import declarative_base from sqlalchemy.orm import sessionmaker from sqlalchemy import Column, create_engine, String db_hostname = os.environ['DB_HOSTNAME'] db_username = os.environ['DB_USERNAME'] db_password = os.environ['DB_PASSWORD'] db_servicename = os.environ['DB_SERVICENAME'] engine_string = f"postgresql://{db_username}:{db_password}@{db_hostname}:5432/{db_servicename}" engine = create_engine(engine_string, isolation_level='REPEATABLE READ', poolclass=sqlalchemy.pool.NullPool ) base = declarative_base() class Testing(base): __tablename__ = 'testing' name = Column(String, primary_key=True) comment = Column(String) base.metadata.create_all(engine) S1 = sessionmaker(engine) with S1() as session: test1 = Testing(name="Jimmy", comment="test1") session.add(test1) session.commit() S2 = sessionmaker(engine, autocommit=True) with S2() as session: test2 = Testing(name="Johnny", comment="test2") session.add(test2)  In this code, the first row with name="Jimmy" and an explicit session.commit() is saved to the table. But the second row with name="Johnny" is not saved. Specifying autocommit=True on the session appears to have no effect. What is the cause?

Answer: If you enable the SQLALCHEMY_WARN_20=1 environment variable you will see  RemovedIn20Warning: The Session.autocommit parameter is deprecated and will be removed in SQLAlchemy version 2.0. …  The "2.0 way" to accomplish that "autocommit" behaviour is to do S2 = sessionmaker(engine) with S2() as session, session.begin(): test2 = Testing(name="Johnny", comment="test2") session.add(test2) # no explicit session.commit() required  The changes will automatically be committed when the context manager (with block) exits, provided that no errors have occurred.
Question: I want to write a function that takes a tuple and returns a tuple of the same size but with each element wrapped in optional. Pseudo code: from typing import TypeVar T = TypeVar("T", bound=tuple[dict[str, str], ...]) def f(tup: T) -> Map[Optional, T]: # Dummy implementation return [None if ... else el for el in tup]  Here Map is a made-up, type-level function that wraps each returned element's type in Optional. Concretely, if the input type was e.g. tuple[dict[str, str], dict[str, str]] I want the return type to be tuple[Optional[dict[str, str]], Optional[dict[str, str]]].

Answer: TLDR: Use @overload to define a practically sufficient number of items and a less strictly typed catch-all case.  There is currently no way to unpack type variables from a tuple and transform each of them. However, for practical usage it is usually sufficient to explicitly type cases for a low number of items. This is done using @overload to collect several explicitly typed cases. A less well-typed catch-all case for an arbitrary number of items retains some type-checking capabilities in any case. from typing import overload, Optional, TypeVar T1 = TypeVar("T1") T2 = TypeVar("T2") T3 = TypeVar("T3") TN = TypeVar("TN") # manually defined cases for usually expected item counts @overload def g(t: tuple[T1]) -> tuple[Optional[T1]]: ... @overload def g(t: tuple[T1, T2]) -> tuple[Optional[T1], Optional[T2]]: ... @overload def g(t: tuple[T1, T2, T3]) -> tuple[Optional[T1], Optional[T2], Optional[T3]]: ... # catch all case when none of the explicit item counts apply @overload def g(t: tuple[TN, ...]) -> tuple[Optional[TN], ...]: ... # actual runtime implementation def g(tup): return tuple(None if ... else el for el in tup)  The number of item count overloads is a trade-off between covering actually needed case and maintaining the largely duplicated signatures; it is common to define up to about half a dozen. The standard library (via typeshed) currently defines up to 5 explicit element types, for example for zip.
Question: I want to save a pandas dataframe to a file with to_string(), but want to left align the column values. With to_string(justify=left), only column labels are left aligned. For example with pd.DataFrame({'col1': [' 123 ', ' 1234'], 'col2': ['1', '444441234']}).to_string(index=False) I get the following result:  I want to get rid of the whitespaces in the first row by left aligning the column values.

Answer: The to_string methods provides support for per column formatters. It allows you to use specific formats for all of some columns. A rather simple way is to create a format and then apply it with a lambda. The only picky part is that to use left formatting, you will have to know the width of the column. For your provided data, you could left align everything with: df = pd.DataFrame({'col1': [' 123 ', ' 1234'], 'col2': ['1', '444441234']}) widths = [4, 9] formats = ['{' + f':<{i}' + '}' for i in widths] print(df.to_string(index=None, col_space=widths, formatters= [(lambda x: fmt.format(x)) for fmt in formats], justify='left'))  to get: col1 col2 123 1 1234 444441234  You could also left align only some columns by using a dict for the formatters parameter: print(df.to_string(index=None, formatters= {'col2': (lambda x: '{:<9}'.format(x))}, justify='left'))  gives: col1 col2 123 1 1234 444441234
Question: Seeing kind of contradictory results: class A: def __init__(self, a: int): pass  The snippet above passes a mypy test, but the one below doesn't. class A: def __init__(self): pass  Any idea why?

Answer: This is documented here. When you have at least one (annotated) argument for a function, it is considered (at least partially) typed. When neither arguments nor return values are annotated, the function is considered untyped. This distinction is important because by default, mypy does not check the bodies of untyped functions at all. This behavior is configurable via check_untyped_defs. Note that the complaint about the missing return type only arises, if you set disallow_untyped_defs (or strict). Otherwise neither of your examples will trigger an error. The __init__ method receives special treatment because people complained that always returns None and they did not want to explicitly write out the return type accordingly. This is why that behavior is so inconsistent. class A: def __init__(self, x: int): # this is fine with `mypy` pass def foo(self, x: int): print("hi") class B: def __init__(self): pass def foo(self): print("hi")  Out of all these methods, only A.__init__ is considered fully typed (because of the implicit None return by __init__). All the other methods will trigger errors with disallow_untyped_defs set:  error: Function is missing a return type annotation [no-untyped-def]  I don't particularly like this approach, but that is the way they decided to handle it.