eshangj/stackoverflow_q_and_a_sample huggingface dataset:
Question: I need to measure similarity between feature vectors using CCA module. I saw sklearn has a good CCA module available: https://scikit-learn.org/stable/modules/generated/sklearn.cross_decomposition.CCA.html In different papers I reviewed, I saw that the way to measure similarity using CCA is to calculate the mean of the correlation coefficients, for example as done in this following notebook example: https://github.com/google/svcca/blob/1f3fbf19bd31bd9b76e728ef75842aa1d9a4cd2b/tutorials/001_Introduction.ipynb How to calculate the correlation coefficients (as shown in the notebook) using sklearn CCA module? from sklearn.cross_decomposition import CCA import numpy as np U = np.random.random_sample(500).reshape(100,5) V = np.random.random_sample(500).reshape(100,5) cca = CCA(n_components=1) cca.fit(U, V) cca.coef_.shape # (5,5) U_c, V_c = cca.transform(U, V) U_c.shape # (100,1) V_c.shape # (100,1)  This is an example of the sklearn CCA module, however I have no idea how to retrieve correlation coefficients from it.

Answer: In reference to the notebook you provided which is a supporting artefact to and implements ideas from the following two papers  "SVCCA: Singular Vector Canonical Correlation Analysis for Deep Learning Dynamics and Interpretability". Neural Information Processing Systems (NeurIPS) 2017 "Insights on Representational Similarity in Deep Neural Networks with Canonical Correlation". Neural Information Processing Systems (NeurIPS) 2018  The authors there calculate 50 = min(A_fake neurons, B_fake neurons) components and plot the correlations between the transformed vectors of each component (i.e. 50). With the help of the below code, using sklearn CCA, I am trying to reproduce their Toy Example. As we'll see the correlation plots match. The sanity check they used in the notebook came very handy - it passed seamlessly with this code as well. import numpy as np from matplotlib import pyplot as plt from sklearn.cross_decomposition import CCA # rows contain the number of samples for CCA and the number of rvs goes in columns X = np.random.randn(2000, 100) Y = np.random.randn(2000, 50) # num of components n_comps = min(X.shape[1], Y.shape[1]) cca = CCA(n_components=n_comps) cca.fit(X, Y) X_c, Y_c = cca.transform(X, Y) # calculate and plot the correlations of all components corrs = [np.corrcoef(X_c[:, i], Y_c[:, i])[0, 1] for i in range(n_comps)] plt.plot(corrs) plt.xlabel('cca_idx') plt.ylabel('cca_corr') plt.show()  Output:  For the sanity check, replace the Y data matrix by a scaled invertible transform of X and rerun the code. Y = np.dot(X, np.random.randn(100, 100))  Output:
Question: I am trying to linearly scale an image so the whole greyscale range is used. This is to improve the lighting of the shot. When plotting the histogram however I don't know how to get the scaled histogram so that its smoother so it's a curve as aspired to discrete bins. Any tips or points would be much appreciated. import cv2 as cv import numpy as np import matplotlib.pyplot as plt img = cv.imread(r'/Users/harold/Documents/Academia/Nottingham Uni/Year 4/ImageProcessing/Imaging_Task_Sheet/PointImage.jpeg', cv.IMREAD_GRAYSCALE) img_s = img/255 img_s = img_s / np.max(img_s) img_s = img_s*255 histogram = cv.calcHist([img], [0], None, [256], [0, 256]) histogram1 = cv.calcHist([img_s.astype('uint8')], [0], None, [256], [0, 256]) plt.figure() plt.title("Grayscale Histogram") plt.xlabel("grayscale value") plt.ylabel("pixels") plt.plot(histogram, label='Original Image') # <- or here plt.plot(histogram1, label='Equalised Image') # <- or here  The histogram produced is:  Which is from this picture: 

Answer: I think what you have in mind is a spline curve that passes through your points. Here is how to do it: import cv2 as cv import numpy as np import matplotlib.pyplot as plt from scipy import interpolate img = cv.imread(r'3NKTJ.jpg', cv.IMREAD_GRAYSCALE) img_s = img/255 img_s = img_s / np.max(img_s) img_s = img_s*255 histogram = cv.calcHist([img], [0], None, [256], [0, 256]) histogram1 = cv.calcHist([img_s.astype('uint8')], [0], None, [256], [0, 256]) x=np.linspace(0,len(histogram1),len(histogram1)) # x: 0 --> 255 with step=1 X=np.where(histogram1>0)[0] # extract bins with non-zero histogram1 values Y=histogram1[X] # the corresponding Y values F=interpolate.splrep(X, Y) # spline representation of (X,Y) Ynew = interpolate.splev(x, F) # calculate interpolated Ynew plt.figure() plt.title("Grayscale Histogram") plt.xlabel("grayscale value") plt.ylabel("pixels") plt.plot(histogram, label='Original Image') # <- or here plt.plot(histogram1, label='Equalised Image') # <- or here plt.plot(x,Ynew, label='spline interpolation of Equalised Image')  Below, the result:  Best regards, Stéphane
Question: I am trying to read a YAML file that has the tag !vault in it. I get the error:  could not determine a constructor for the tag '!vault'  Upon reading a couple of blogs, I understood that I need to specify some constructors to resolve this issue, but I'm not clear on how to do it. import yaml from yaml.loader import SafeLoader with open('test.yml' ) as stream: try: inventory_info = yaml.safe_load(stream) except yaml.YAMLError as exc: print(exc) User = inventory_info['all']['children']['linux']['vars']['user'] key = inventory_info['all']['children']['linux']['vars']['key_file']  The YAML file I am using: all: children: rhel: hosts: 172.18.144.98 centos: hosts: 172.18.144.98 linux: children: rhel: centos: vars: user: "o9ansibleuser" key_file: "test.pem" ansible_password: !vault | $ANSIBLE_VAULT;2.1;AES256 3234567899353936376166353 

Answer: Either use the from_yaml utility function: from ansible.parsing.utils.yaml import from_yaml # inventory_info = yaml.safe_load(stream) # Change this inventory_info = from_yaml(stream) # to this  Or add the constructor to yaml.SafeLoader: from ansible.parsing.yaml.objects import AnsibleVaultEncryptedUnicode def construct_vault_encrypted_unicode(loader, node): value = loader.construct_scalar(node) return AnsibleVaultEncryptedUnicode(value) yaml.SafeLoader.add_constructor(u'!vault', construct_vault_encrypted_unicode) inventory_info = yaml.safe_load(stream)
Question: I am writing a simple function that sends messages based on a schedule using AsyncIOScheduler. scheduler = AsyncIOScheduler() scheduler.add_job(job, "cron", day_of_week="mon-fri", hour = "16") scheduler.start()  It seems to work, but I always get the following message: PytzUsageWarning: The zone attribute is specific to pytz's interface; please migrate to a new time zone provider. For more details on how to do so, see https://pytz-deprecation-shim.readthedocs.io/en/latest/migration.html PytzUsageWarning: The localize method is no longer necessary, as this time zone supports the fold attribute (PEP 495)  I am not familiar with time in python at all, so I am not really sure what this message is asking me to do. I visited the link provided in the message, but the explanation there is too complicated for me. As I understand, I have to migrate to using PEP495, but how exactly can I do just that?

Answer: To set a PIP495 compatible timezone in APScheduler, set a parameter when instantiating the scheduler: scheduler = AsyncIOScheduler(timezone="Europe/Berlin") scheduler.add_job(job, "cron", day_of_week="mon-fri", hour = "16") scheduler.start()  With flask-APScheduler (version 1.12.2), add the timezone to the configuration class: """Basic Flask Example from flask-apscheduler examples, file jobs.py""" from flask import Flask from flask_apscheduler import APScheduler class Config: """App configuration.""" JOBS = [ { "id": "job1", "func": "jobs:job1", "args": (1, 2), "trigger": "interval", "seconds": 10, } ] SCHEDULER_API_ENABLED = True SCHEDULER_TIMEZONE = "Europe/Berlin" # <========== add here def job1(var_one, var_two): print(str(var_one) + " " + str(var_two)) if __name__ == "__main__": app = Flask(__name__) app.config.from_object(Config()) scheduler = APScheduler() scheduler.init_app(app) scheduler.start() app.run()
Question: I am writing a simple test code in Python using the Hypothesis package. It there a way to use multiple strategies for the same function parameter? As an example, use integers() and floats() to test my values parameter without writing two separate test functions? from hypothesis import given from hypothesis.strategies import lists, integers, floats, sampled_from @given( integers() ,floats() ) def test_lt_operator_interval_bin_numerical(values): interval_bin = IntervalBin(10, 20, IntervalClosing.Both) assert (interval_bin < values) == (interval_bin.right < values)  The above code doesn't work, but it represents what i would like to achieve. NB: I have already tried the trivial solution of creating two different tests with two different strategies: def _test_lt(values): interval_bin = IntervalBin(10, 20, IntervalClosing.Both) assert (interval_bin < values) == (interval_bin.right < values) test_lt_operator_interval_bin_int = given(integers())(_test_lt) test_lt_operator_interval_bin_float = given(floats())(_test_lt)  However I was wondering if there was a nicer way to do this: when the nummber of strategies becomes larger, it is quite redundant as code.

Answer: In general if you need your values to be one of several things (like ints or floats in your example) then we can combine strategies for separate things into one with | operator (which operates similar to one for sets -- a union operator and works by invoking __or__ magic method): from hypothesis import given from hypothesis.strategies import integers, floats @given( integers() | floats() ) def test_lt_operator_interval_bin_numerical(values): interval_bin = IntervalBin(10, 20, IntervalClosing.Both) assert (interval_bin < values) == (interval_bin.right < values)  or you can use strategies.one_of as @Zac Hatfield-Dodds mentioned which does pretty much the same thing: from hypothesis import given from hypothesis.strategies import integers, floats, one_of @given( one_of(integers(), floats()) ) def test_lt_operator_interval_bin_numerical(values): interval_bin = IntervalBin(10, 20, IntervalClosing.Both) assert (interval_bin < values) == (interval_bin.right < values)
Question: Core metadata specification documents the metadata field Requires-External which seems to be for specifying system (non-python) dependencies. How do you actually specify this field though? This is what I've tried: . ├── mypackage │ └── __init__.py └── setup.py  Contents of setup.py from setuptools import setup setup( name="mypackage", description="blah blah", url='https://example.org', version="0.1", packages=["mypackage"], requires_external=[ "C", "libpng (>=1.5)", 'make; sys_platform != "win32"', ], )  When I build this package, that metadata was not included Metadata-Version: 2.1 Name: mypackage Version: 0.1 Summary: blah blah Home-page: https://example.org License: UNKNOWN Platform: UNKNOWN UNKNOWN  So what is the syntax to pass Requires-External to setuptools/distutils? Note: this question is not asking about Requires-Dist metadata.

Answer:  So what is the syntax to pass Requires-External to setuptools/distutils?  There is none by default, as neither distutils nor setuptools support the field. Also, requires_external keyword arg is not supported as well - it is silently ignored, just as any other unknown keyword arg. To add local support for requires_external kwarg, you need to provide your own distribution implementation. Minimal example (requires Python 3.9): from setuptools.dist import Distribution as DistributionOrig class Distribution(DistributionOrig): _DISTUTILS_UNSUPPORTED_METADATA = DistributionOrig._DISTUTILS_UNSUPPORTED_METADATA | {'requires_external': dict}  You can now pass requires_external to setup() when paired with distclass: setup( ..., requires_external=[ 'C', 'libpng (>=1.5)', 'make; sys_platform != "win32"', ], distclass=Distribution, )  The next goal is to actually write the contents of requires_external to PKG-INFO. This is quite tricky to do via distribution metadata itself because setuptools patches the relevant methods (DistributionMetadata.read_pkg_file() and DistributionMetadata.write_pkg_file()) with own monolithic impls. The easiest way for this IMO is to modify PKG-INFO post factum via a custom egg_info impl: import email from pathlib import Path from setuptools.command.egg_info import egg_info as egg_info_orig class egg_info(egg_info_orig): def run(self): super().run() # PKG-INFO is now guaranteed to exist pkg_info_file = Path(self.egg_info, 'PKG-INFO') pkg_info = email.message_from_bytes(pkg_info_file.read_bytes()) for req in self.distribution.metadata.requires_external: pkg_info.add_header('Requires-External', req) pkg_info_file.write_bytes(pkg_info.as_bytes())  Pass your own egg_info impl via cmdclass argument in setup(): setup( ..., requires_external=[ 'C', 'libpng (>=1.5)', 'make; sys_platform != "win32"', ], distclass=Distribution, cmdclass={'egg_info': egg_info}, )
Question: I have run the sklearn.manifold.TSNE example code from the sklearn documentation, but I got the error described in the questions' title. I have already tried updating my sklearn version to the latest one (by !pip install -U scikit-learn) (scikit-learn=1.0.1). However, the problem is still there. Does anyone know how to fix it?  python = 3.7.12 sklearn= 1.0.1  Example code: import numpy as np from sklearn.manifold import TSNE X = np.array([[0, 0, 0], [0, 1, 1], [1, 0, 1], [1, 1, 1]]) X_embedded = TSNE(n_components=2, learning_rate='auto', init='random').fit_transform(X) X_embedded.shape  The error line happened in: X_embedded = TSNE(n_components=2, learning_rate='auto', init='random').fit_transform(X)  Error message: UFuncTypeError: ufunc 'multiply' did not contain a loop with signature matching types (dtype('<U32'), dtype('<U32')) -> dtype('<U32') 

Answer: Delete learning_rate='auto' solved my problem. Thanks @FlaviaGiammarino comment!!
Question: I have a list of sets like this: set_list = [{1, 2, 3}, {4, 5, 1, 6}, {2, 3, 6}, {1, 5, 8}]  Now I want to merge all of the sets together and return a set of all sets like this: final_set = {1, 2, 3, 4, 5, 6, 8}  I have used this code but it is not working correctly: tmp_list = [] final_set = set(tmp_list.append(elem) for elem in set_list)  What should I do?

Answer: You can use unpacking with set().union for a clean one-liner. >>> set().union(*set_list) {1, 2, 3, 4, 5, 6, 8}
Question: I have a function that reads and handles *.csv files in several dataframes. However, not all of the .csv files have the same separator. Is there a way to detect which type of separator the .csv file has, and then use it in the Pandas' read_csv() function? df = pd.read_csv(path, sep = 'xxx',header = None, index_col = 0) 

Answer: Update In fact, use engine='python' as parameter of read_csv. It will try to automatically detect the right delimiter.  sepstr, default ‘,’ Delimiter to use. If sep is None, the C engine cannot automatically detect the separator, but the Python parsing engine can, meaning the latter will be used and automatically detect the separator by Python’s builtin sniffer tool, csv.Sniffer. In addition, separators longer than 1 character and different from '\s+' will be interpreted as regular expressions and will also force the use of the Python parsing engine. Note that regex delimiters are prone to ignoring quoted data. Regex example: '\r\t'.  Use csv.Sniffer: import csv def find_delimiter(filename): sniffer = csv.Sniffer() with open(filename) as fp: delimiter = sniffer.sniff(fp.read(5000)).delimiter return delimiter  Demo: >>> find_delimiter('data.csv') ',' >>> find_delimiter('data.txt') ' '
Question: I have done the following code but do not understand properly what is going on there. Can anyone explain how to fill colors in Numpy? Also I want to set in values in a way from 1 to 0 to give spectrum an intensity. E.g-: 0 means low intensity, 1 means high intensity import numpy as np import matplotlib.pyplot as plt a= np.zeros([256*6,256*6, 3], dtype=np.uint8) # init the array # fill the array with rgb values to create the spectrum without the use of loops #red a[:,:,0] = np.concatenate(([255]*256, np.linspace(255,0,256), [0]*256, [0]*256, np.linspace(0,255,256), [255]*256)) #green a[:,:,1] = np.concatenate((np.linspace(0,255,256), [255]*256, [255]*256, np.linspace(255,0,256), [0]*256,[0]*256)) #blue a[:,:,2] = np.concatenate(([0]*256, [0]*256,np.linspace(0,255,256),[255]*256, [255]*256, np.linspace(255,0,256))) plt.imshow(a) # this is different than what I am looking for  Expected Output-: 

Answer:  res=256 op=np.zeros([res,res, 3]) # init the array #RGB op[:,:,0]= np.linspace(0,1,res) op[:,:,1]=np.linspace(0,1,res).reshape (256, 1) op[:,:,2]= np.linspace(1,0,res) plt.imshow(op)  it will give the exact thing that you are looking for! let me know if it does not work
Question: Code of .ipynb file:  Python is detected:  The code has color in jupyter notebook:  I tried setting up a jupyter notebook in vs code in an anaconda environment. I have tried - Python: Select interpreter and selected my anaconda environment. Made sure python is in the environment: python --version: Python 3.8.8. Tried clicking on CVE(in the bottom right corner) to change to python (picture 2). The colorization of the code works fine in the interactive window of jupyter notebook (picture 3).

Answer: Jupyter extension detects your code as CVE instead of Python so Python syntax highlighting is not applied successfully. Refer to Jupyter in vscode can't execute syntax highlighting, the Dependency Analytics extension should be the reason. Remove or Disable it then reload window, the question should go away.
Question: Let's say I have a GeoDataFrame with a CRS set. gdf.crs  gives me <Projected CRS: EPSG:25833> Name: ETRS89 / UTM zone 33N Axis Info [cartesian]: - [east]: Easting (metre) - [north]: Northing (metre) Area of Use: - undefined Coordinate Operation: - name: UTM zone 33N - method: Transverse Mercator Datum: European Terrestrial Reference System 1989 - Ellipsoid: GRS 1980 - Prime Meridian: Greenwich  This is of type <class 'pyproj.crs.crs.CRS'>. Is there a way to extract the EPSG Code from this, thus 25833?

Answer: .crs returns a pyroj.CRS object. This should get you the EPSG code from the object: gdf.crs.to_epsg()  pyproj docs
Question: I'm just starting to create an image classification program in Python with TensorFlow using the CIFAR10 dataset, following this tutorial. Here is my code so far: import tensorflow as tf from tensorflow.keras import datasets, layers, models import matplotlib.pyplot as plt import numpy as np datasets.cifar10.load_data()  I'm getting an error as follows:  File "C:\Users\Admin\AppData\Local\Programs\Python\Python39\lib\urllib\request.py", line 1346, in do_open h.request(req.get_method(), req.selector, req.data, headers, File "C:\Users\Admin\AppData\Local\Programs\Python\Python39\lib\http\client.py", line 1257, in request self._send_request(method, url, body, headers, encode_chunked) File "C:\Users\Admin\AppData\Local\Programs\Python\Python39\lib\http\client.py", line 1303, in _send_request self.endheaders(body, encode_chunked=encode_chunked) File "C:\Users\Admin\AppData\Local\Programs\Python\Python39\lib\http\client.py", line 1252, in endheaders self._send_output(message_body, encode_chunked=encode_chunked) File "C:\Users\Admin\AppData\Local\Programs\Python\Python39\lib\http\client.py", line 1012, in _send_output self.send(msg) File "C:\Users\Admin\AppData\Local\Programs\Python\Python39\lib\http\client.py", line 952, in send self.connect() File "C:\Users\Admin\AppData\Local\Programs\Python\Python39\lib\http\client.py", line 1426, in connect self.sock = self._context.wrap_socket(self.sock, File "C:\Users\Admin\AppData\Local\Programs\Python\Python39\lib\ssl.py", line 500, in wrap_socket return self.sslsocket_class._create( File "C:\Users\Admin\AppData\Local\Programs\Python\Python39\lib\ssl.py", line 1040, in _create self.do_handshake() File "C:\Users\Admin\AppData\Local\Programs\Python\Python39\lib\ssl.py", line 1309, in do_handshake self._sslobj.do_handshake() ssl.SSLCertVerificationError: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: certificate has expired (_ssl.c:1129)  What does this error mean, and what can I do to fix it? I'm a beginner with TensorFlow and haven't been able to find an exact solution for this issue.

Answer: Recently had a similar error using python 3.7 and just turned off the verification like this: import tensorflow as tf from tensorflow.keras import datasets, layers, models import matplotlib.pyplot as plt import numpy as np import ssl ssl._create_default_https_context = ssl._create_unverified_context datasets.cifar10.load_data()  Check out this issue for more information. You can of course try to install the missing certificates for your Python version. I can confirm that your code works on Google Colab. So it is definitely due to some missing certificates.
Question: Following code is borrowed from here: import numpy as np import matplotlib.cm as cm import matplotlib.pyplot as plt delta = 0.025 x = np.arange(-3.0, 3.0, delta) y = np.arange(-2.0, 2.0, delta) X, Y = np.meshgrid(x, y) Z1 = np.exp(-X**2 - Y**2) Z2 = np.exp(-(X - 1)**2 - (Y - 1)**2) Z = (Z1 - Z2) * 2 fig, ax = plt.subplots() CS = ax.contour(X, Y, Z) ax.clabel(CS, inline=True, fontsize=10) ax.set_title('Simplest default with labels')  The above code produces the contour plot with 7 contour lines. How to draw a single contour line say with the value 0.5? The output of the above code is: 

Answer: Just add contour value: CS = ax.contour(X, Y, Z, [0.5]) import numpy as np import matplotlib.cm as cm import matplotlib.pyplot as plt delta = 0.025 x = np.arange(-3.0, 3.0, delta) y = np.arange(-2.0, 2.0, delta) X, Y = np.meshgrid(x, y) Z1 = np.exp(-X**2 - Y**2) Z2 = np.exp(-(X - 1)**2 - (Y - 1)**2) Z = (Z1 - Z2) * 2 fig, ax = plt.subplots() CS = ax.contour(X, Y, Z, [0.5]) ax.clabel(CS, inline=True, fontsize=10) ax.set_title('Simplest default with labels') plt.show()
Question: I use pre-commit to run black flake8 and isort on my code. I ran pre-commit install and as expected it created .git/hooks/pre-commit which starts like: #!/usr/bin/env python3.9 # File generated by pre-commit: https://pre-commit.com # ID: 138fd403232d2ddd5efb44317e38bf03 import os import sys ...  The hook works fine in the terminal: $ git commit -am "remove commented block" isort....................................................................Passed black....................................................................Passed flake8...................................................................Passed [main f30007d] remove commented block 1 file changed, 4 deletions(-)  But running it from VSCode's Source Control Panel yields an error (command output): > git -c user.useConfigOnly=true commit --quiet --allow-empty-message --file - env: python3.9: No such file or directory  Not sure where that comes from. In addition (not sure it matters though) I double checked: both the terminal's python and VSCode's selected Python interpreter point to the same /Users/victor/.pyenv/shims/python

Answer: It seemed to be an issue with pyenv not being loaded by VSCode's source control panel when executing the git commands. I tried moving some stuff (like $(pyenv init -)) into an earlier zsh configuration file like .zshenv but that did not help. In the end, specifying the complete path fixed it #!/usr/bin/env /Users/victor/.pyenv/shims/python3.9 # File generated by pre-commit: https://pre-commit.com # ID: 138fd403232d2ddd5efb44317e38bf03 import os import sys ...