eshangj/stackoverflow_q_and_a_sample huggingface dataset:
Question: One can use Julia's built-in modules and functions using the juliacall package. for example: >>> from juliacall import Main as jl >>> import numpy as np # Create a 2*2 random matrix >>> arr = jl.rand(2,2) >>> arr <jl [0.28133223988783074 0.22498491616860727; 0.008312971104033062 0.12927167014532326]> # Check whether Numpy can recognize the shape of the array or not >>> np.array(arr).shape (2, 2) >>> type(np.array(arr)) <class 'numpy.ndarray'>  Then I'm curious to know if importing an installed julia package into Python is possible? For example, let's say that one wants to import Flux.jl into Python. Is there a way to achieve this?

Answer: I found it by scrutinizing the pictures of the second example of the juliacall GitHub page. According to the example, I'm able to import Flux.jl by taking these steps: >>> from juliacall import Main as jl >>> jl.seval("using Flux")  Also, one can install any registered Julia package using Pkg in Python: >>> from juliacall import Main as jl >>> jl.Pkg.add("Flux") # After a successful installation >>> jl.Pkg.status() Status `C:\Users\Shayan\miniconda3\envs\im\julia_env\Project.toml` [587475ba] Flux v0.14.10 [6099a3de] PythonCall v0.9.15 # This indicates that the installation was successful  Additional note One can use Julia packages in Python using the pyjulia package that can be installed using pip install julia in Python. Afterward, the following commands should be entered: >>> import julia >>> julia.install() # Then >>> from julia import Pkg >>> Pkg.add("Flux") # After a successful installation >>> Pkg.status() Status `C:\Users\Shayan\.julia\environments\v1.10\Project.toml` [587475ba] Flux v0.14.10 # This indicates that the installation was successful >>> from julia import Flux  You can also say, from julia import Flux as flx. Note that using juliacall is recommended.
Question: The standard method that I know to get number of cores in python is to use multiprocess.cpu_count import multiprocessing print(multiprocessing.cpu_count())  Also, when creating a task we can specify some cores that the process can access using taskset. Apparently cpu_count is always getting the number of available cores of the machine, regardless if the process was restricted to use only some of those cores. python nproc.py taskset 7 python nproc.py taskset 1 python nproc.py  nproc on the other hand gives the number of cores available for the process. taskset 7 nproc # gives at most 3 taskset 1 nproc # gives 1  I know I could invoke nproc or taskset -p {os.getpid()} to get the correct number of cores. Is there another way to do this, without relying on other programs (reading /proc/{os.pid()}/* could be a solution). Thank you

Answer: According to the docs, this may be of limited availability, but it seems the os library has what you want;  Interface to the scheduler These functions control how a process is allocated CPU time by the operating system. They are only available on some Unix platforms. For more detailed information, consult your Unix manpages. New in version 3.3. ... os.sched_getaffinity(pid) Return the set of CPUs the process with PID pid (or the current process if zero) is restricted to.  tested using Ubuntu 20.04.2 (WSL): aaron@DESKTOP:/mnt/c$ python3 -c "import os; print(os.sched_getaffinity(0))" {0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11} aaron@DESKTOP:/mnt/c$ taskset 7 python3 -c "import os; print(os.sched_getaffinity(0))" {0, 1, 2}  Edit: on windows you'll probably have to look to psutil
Question: I am working on a project that uses a lot of xml, and would like to use pydantic to model the objects. In this case I simplified the xml but included an example object. <ns:SomeType name="NameType" shortDescription="some data"> <ns:Bar thingOne="alpha" thingTwo="beta" thingThree="foobar"/> </ns:SomeType>  Code from pydantic import BaseModel from typing import Optional, List from xml.etree import ElementTree as ET class Bar(BaseModel): thing_one: str thing_two: str thing_three: str class SomeType(BaseModel): name: str short_description: str bar: Optional[Bar] def main(): with open("path/to/file.xml") as fp: source = fp.read() root = ET.fromstring(source) some_type_list = [] for child in root: st = SomeType( name=child.attrib["name"], short_description=child.attrib["shortDescription"], ) for sub in child: st.bar = Bar( thing_one=sub.attrib["thingOne"], thing_two=sub.attrib["thingTwo"], thing_three=sub.attrib["thingThree"], )  I looked into BaseModel.parse_obj or BaseModel.parse_raw but I don't think that will solve the problem. I also thought I could try to use xmltodict to convert the xml, the namespace's and the @ attribute's get even more in the way... >>> import xmltodict >>> xmltodict.parse(input_xml) {'ns:SomeType': {'@name': 'NameType', '@shortDescription': 'some data', ... }} 

Answer: xmltodict can help in your example if you combine it with field aliases: from typing import Optional import xmltodict from pydantic import BaseModel, Field class Bar(BaseModel): thing_one: str = Field(alias="@thingOne") thing_two: str = Field(alias="@thingTwo") thing_three: str = Field(alias="@thingThree") class SomeType(BaseModel): name: str = Field(alias="@name") short_description: str = Field(alias="@shortDescription") bar: Optional[Bar] = Field(alias="ns:Bar") class Root(BaseModel): some_type: SomeType = Field(alias="ns:SomeType") print( Root.model_validate( xmltodict.parse( """<ns:SomeType name="NameType" shortDescription="some data"> <ns:Bar thingOne="alpha" thingTwo="beta" thingThree="foobar"/> </ns:SomeType>""")).some_type)  Output: name='NameType' short_description='some data' bar=Bar(thing_one='alpha', thing_two='beta', thing_three='foobar')  You can see in the example above that a Root model is needed because the dict has an ns:SomeType key.
Question: I can't find any Python code for the equivalent of python -m http.server port --bind addr --directory dir  So I need basically a working server class that process at least GET requests. Most of the things I found on Google were either an HTTP server with some special needs or something like that, where you need to code the response behaviour be yourself: from http.server import BaseHTTPRequestHandler, HTTPServer def run(server_class=HTTPServer, handler_class=BaseHTTPRequestHandler): server_address = ('', 8000) httpd = server_class(server_address, handler_class) httpd.serve_forever() run()  All that I need is a default working skeleton of a Python HTTP server, where you can provide address, port and directory, and it would normally process GET requests.

Answer: That's what I ended up with: # python -m http.server 8000 --directory ./my_dir from http.server import HTTPServer as BaseHTTPServer, SimpleHTTPRequestHandler import os class HTTPHandler(SimpleHTTPRequestHandler): """This handler uses server.base_path instead of always using os.getcwd()""" def translate_path(self, path): path = SimpleHTTPRequestHandler.translate_path(self, path) relpath = os.path.relpath(path, os.getcwd()) fullpath = os.path.join(self.server.base_path, relpath) return fullpath class HTTPServer(BaseHTTPServer): """The main server, you pass in base_path which is the path you want to serve requests from""" def __init__(self, base_path, server_address, RequestHandlerClass=HTTPHandler): self.base_path = base_path BaseHTTPServer.__init__(self, server_address, RequestHandlerClass) web_dir = os.path.join(os.path.dirname(__file__), 'my_dir') httpd = HTTPServer(web_dir, ("", 8000)) httpd.serve_forever()  A simple HTTP server that handles GET requests with, working with a certain directory.
Question: I want to be able to install higher versions of R in my conda environment. In particular r 4.1.2. I have also installed Mamba fyi. Currently r-base has: conda activate main conda search r-base Name Version r-base 3.2.4 r-base 3.3.0 r-base 3.3.1 r-base 3.3.2 r-base 3.4.1 r-base 3.4.2 r-base 3.4.3 r-base 3.4.3 r-base 3.5.1 r-base 3.5.3 r-base 3.6.0 r-base 3.6.1 r-base 3.6.1 

Answer: You could specify the version directly: mamba install -c conda-forge r-base=4.1.2 or conda install -c conda-forge r-base=4.1.2 On Jul 25, 2022 the highest available in conda-forge was 4.1.3 while in pkgs/r 4.2.0 was the highest version available.
Question: I got the following message after running my Flask project on another system. The application ran all the time without problems: Error: While importing 'app', an ImportError was raised: Traceback (most recent call last): File "c:\users\User\appdata\local\programs\python\python39\lib\site-packages\flask\cli.py", line 214, in locate_app __import__(module_name) File "C:\Users\User\Desktop\Projekt\app\__init__.py", line 3, in <module> from flask_restx import Namespace, Api File "c:\users\User\appdata\local\programs\python\python39\lib\site-packages\flask_restx\__init__.py", line 5, in <module> File "c:\users\User\appdata\local\programs\python\python39\lib\site-packages\flask_restx\api.py", line 50, in <module> from .swagger import Swagger File "c:\users\User\appdata\local\programs\python\python39\lib\site-packages\flask_restx\swagger.py", line 18, in <module> from werkzeug.routing import parse_rule ImportError: cannot import name 'parse_rule' from 'werkzeug.routing' (c:\users\User\appdata\local\programs\python\python39\lib\site-packages\werkzeug\routing\__i nit__.py)  My requirements.txt Flask~=2.1.2 psycopg2-binary==2.9.3 Flask-SQLAlchemy==2.5.1 flask-restx==0.5.1 qrcode~=7.3.1 PyPDF2==2.6.0 reportlab~=3.6.10 WTForms~=3.0.1 flask-bootstrap==3.3.7.1 flask-wtf==1.0.1 

Answer: The workaround I use for now is to pin werkzeug to 2.1.2 in requirements.txt. This should only be done until the other libraries are compatible with the latest version of Werkzeug, at which point the pin should be updated. werkzeug==2.1.2
Question: Simple demo application I am trying to set the theme to dark. I would prefer a code version (non QtQuick preferred), but only way I see for Python is with a QtQuick config file, and even that does not work. from PySide6 import QtWidgets from PySide6 import QtQuick if __name__ == '__main__': app = QtWidgets.QApplication() app.setApplicationDisplayName("Should be Dark Theme") app.setStyle("Universal") view = QtQuick.QQuickView() view.show() app.exec()  And I have a qtquickcontrols2.conf configuration file in the same directory. (Also tried setting QT_QUICK_CONTROLS_CONF to absolute path.) [Controls] Style=Material [Universal] Theme=Dark [Material] Theme=Dark  And yet, it's still bright white:  I do not care if it is Material or Universal style, just want some built in dark mode for the title bar. In the end, need a way to make the titlebar dark without creating a custom one. Thank you for any guidance!

Answer: import sys sys.argv += ['-platform', 'windows:darkmode=2'] app = QApplication(sys.argv)  above 3 lines can change your window to dark mode if you are using windows and Fusion style makes the app more beautiful, tested in windows 10, 11 example:- from PySide6.QtWidgets import ( QApplication, QCheckBox, QComboBox, QDateEdit, QDateTimeEdit, QDial, QDoubleSpinBox, QFontComboBox, QLabel, QLCDNumber, QLineEdit, QMainWindow, QProgressBar, QPushButton, QRadioButton, QSlider, QSpinBox, QTimeEdit, QVBoxLayout, QWidget, ) import sys sys.argv += ['-platform', 'windows:darkmode=2'] class MainWindow(QMainWindow): def __init__(self): super().__init__() self.setWindowTitle("Widgets App") layout = QVBoxLayout() widgets = [ QCheckBox, QComboBox, QDateEdit, QDateTimeEdit, QDial, QDoubleSpinBox, QFontComboBox, QLCDNumber, QLabel, QLineEdit, QProgressBar, QPushButton, QRadioButton, QSlider, QSpinBox, QTimeEdit, ] for w in widgets: layout.addWidget(w()) widget = QWidget() widget.setLayout(layout) self.setCentralWidget(widget) app = QApplication(sys.argv) app.setStyle('Fusion') window = MainWindow() window.show() app.exec()
Question: Issue I've got a problem using poetry install in my CI/CD pipeline (Github Actions), on any GitHub runner, since I migrated from Python 3.8 to Python 3.10. Installing dependencies from lock file Package operations: 79 installs, 0 updates, 0 removals • Installing pyparsing (3.0.9) JSONDecodeError Expecting value: line 1 column 1 (char 0) at /opt/hostedtoolcache/Python/3.10.0/x64/lib/python3.10/json/decoder.py:355 in raw_decode 351│ """ 352│ try: 353│ obj, end = self.scan_once(s, idx) 354│ except StopIteration as err: → 355│ raise JSONDecodeError("Expecting value", s, err.value) from None 356│ return obj, end 357│ Error: Process completed with exit code 1.  I didn't change any lib configuration in my pyproject.toml file, but as you can see above: Poetry hides most of the StackTrace. What I tried  Recreating the poetry.lock file. Removing the Poetry cache using rm -r ~/.cache/pypoetry/cache/ (and rm -r ~/.cache/pypoetry/). Removing the lib that was returning an error (actually, the issue seems to happen with any lib, so that's the reason why I understand it's probably related to poetry and python)  Question Any idea how to resolve this issue in my CI/CD pipeline?

Answer: After a few researches, I found this thread on the poetry GitHub repository from november 2021. There is this workaround from hoefling GitHub user:  Disabling poetry's experimental new installer may be a workaround for now:  Solution poetry config experimental.new-installer false  Adding this line in the shell before running the poetry install command resolved my problem!   Note that in the same thread, another comment from ddc67cd stated that:  the issue is resolved with the new version of cachecontrol==0.12.9 (it should be installed automatically).  But running pip install -U cachecontrol didn't resolve the issue in my specific case (might be worth testing otherwise?).   It also seems the problem came back recently (July 2022) and this comment suggested a possible root cause to the issue related to the setuptools library. Anyway, disabling poetry's experimental new installer should resolve the problem for now, until a permanent solution is found.
Question: When a user uploads an image from the Django admin panel, I want to change the image format to '.webp'. I have overridden the save method of the model. Webp file is generated in the media/banner folder but the generated file is not saved in the database. How can I achieve that? def save(self, *args, **kwargs): super(Banner, self).save(*args, **kwargs) im = Image.open(self.image.path).convert('RGB') name = 'Some File Name with .webp extention' im.save(name, 'webp') self.image = im  But After saving the model, instance of the Image class not saved in the database? My Model Class is : class Banner(models.Model): image = models.ImageField(upload_to='banner') device_size = models.CharField(max_length=20, choices=Banner_Device_Choice) 

Answer: from django.core.files import ContentFile  If you already have the webp file, read the webp file, put it into the ContentFile() with a buffer (something like io.BytesIO). Then you can proceed to save the ContentFile() object to a model. Do not forget to update the model field, and save the model! https://docs.djangoproject.com/en/4.1/ref/files/file/ Alternatively  "django-webp-converter is a Django app which straightforwardly converts static images to WebP images, falling back to the original static image for unsupported browsers."  It might have some save capabilities too. https://django-webp-converter.readthedocs.io/en/latest/ The cause You are also saving in the wrong order, the correct order to call the super().save() is at the end. Edited, and tested solution: from django.core.files import ContentFile from io import BytesIO def save(self, *args, **kwargs): #if not self.pk: #Assuming you don't want to do this literally every time an object is saved. img_io = BytesIO() im = Image.open(self.image).convert('RGB') im.save(img_io, format='WEBP') name="this_is_my_webp_file.webp" self.image = ContentFile(img_io.getvalue(), name) super(Banner, self).save(*args, **kwargs) #Not at start anymore
Question: I'm trying to create an Instagram bot using InstaPy. I'm following this tutorial. When I ran: from instapy import InstaPy session = InstaPy(username="your username", password="your password") session.login()  I got this error: ImportError: cannot import name 'UNICODE_EMOJI' from 'emoji.unicode_codes' (C:\Users\roeegg22\AppData\Local\Programs\Python\Python310\lib\site-packages\emoji\unicode_codes\__init__.py)  I tried some solutions online, but none of them worked.

Answer: This happens because instapy (or some other library) doesn't reflect the latest update to the emoji library. You should be able to fix it by running pip uninstall emoji pip install emoji==1.7  in the terminal. That way you install the version of emoji library that instapy is made around and the import should work.
Question: How can I create the matrix [[a, 0, 0], [0, a, 0], [0, 0, a], [b, 0, 0], [0, b, 0], [0, 0, b], ...]  from the vector [a, b, ...]  efficiently? There must be a better solution than np.squeeze(np.reshape(np.tile(np.eye(3), (len(foo), 1, 1)) * np.expand_dims(foo, (1, 2)), (1, -1, 3)))  right?

Answer: You can create a zero array in advance, and then quickly assign values by slicing: def concated_diagonal(ar, col): ar = np.asarray(ar).ravel() size = ar.size ret = np.zeros((col * size, col), ar.dtype) for i in range(col): ret[i::col, i] = ar return ret  Test: >>> concated_diagonal([1, 2, 3], 3) array([[1, 0, 0], [0, 1, 0], [0, 0, 1], [2, 0, 0], [0, 2, 0], [0, 0, 2], [3, 0, 0], [0, 3, 0], [0, 0, 3]])  Note that because the number of columns you require is small, the impact of the relatively slow Python level for loop is acceptable: %timeit concated_diagonal(np.arange(1_000_000), 3) 17.1 ms ± 84.3 µs per loop (mean ± std. dev. of 7 runs, 100 loops each)  Update: A solution with better performance! This is done in one step by clever reshaping and slice assignment: def concated_diagonal(ar, col): ar = np.asarray(ar).reshape(-1, 1) size = ar.size ret = np.zeros((col * size, col), ar.dtype) ret.reshape(size, -1)[:, ::col + 1] = ar return ret  Time comparison: %timeit concated_diagonal(np.arange(1_000_000), 3) 10.7 ms ± 198 µs per loop (mean ± std. dev. of 7 runs, 100 loops each)
Question: I'm trying to create hive/impala tables base on avro files in HDFS. The tool for doing the transformations is Spark. I can't use spark.read.format("avro") to load the data into a dataframe, as in that way the doc part (description of the column) will be lost. I can see the doc by doing:  input = sc.textFile("/path/to/avrofile") avro_schema = input.first() # not sure what type it is  The problem is, it's a nested schema and I'm not sure how to traverse it to map the doc to the column description in dataframe. I'd like to have doc to the column description of the table. For example, the input schema looks like: "fields": [ { "name":"productName", "type": [ "null", "string" ], "doc": "Real name of the product" "default": null }, { "name" : "currentSellers", "type": [ "null", { "type": "record", "name": "sellers", "fields":[ { "name": "location", "type":[ "null", { "type": "record" "name": "sellerlocation", "fields": [ { "name":"locationName", "type": [ "null", "string" ], "doc": "Name of the location", "default":null }, { "name":"locationArea", "type": [ "null", "string" ], "doc": "Area of the location",#The comment needs to be added to table comments "default":null .... #These are nested fields  In the final table, for example one field name would be currentSellers_locationName, with column description "Name of the location". Could someone please help to shed some light on how to parse the schema and add the doc to description? and explain a bit about what this below bit is about outside of the fields? Many thanks. Let me know if I can explain it better.  "name" : "currentSellers", "type": [ "null", { "type": "record", "name": "sellers", "fields":[ { 

Answer: If you would like to parse the schema yourself and manually add metadata to spark, I would suggest flatdict package: from flatdict import FlatterDict flat_schema = FlatterDict(schema) # schema as python dict names = {k.replace(':name', ''): flat_schema[k] for k in flat_schema if k.endswith(':name')} docs = {k.replace(':doc', ''): flat_schema[k] for k in flat_schema if k.endswith(':doc')} # keep only keys which are present in both names and docs keys_with_doc = set(names.keys()) & set(docs.keys()) full_name = lambda key: '_'.join( names[k] for k in sorted(names, key=len) if key.startswith(k) and k.split(':')[-2] == 'fields' ) name_doc_map = {full_name(k): docs[k] for k in keys_with_doc}  A typical set of keys in flat_schema.keys() is: 'fields:1:type:1:fields:0:type:1:fields:0:type:1', 'fields:1:type:1:fields:0:type:1:fields:0:name', 'fields:1:type:1:fields:0:type:1:fields:0:default', 'fields:1:type:1:fields:0:type:1:fields:0:doc',  These strings can now be manipulated:  extract only the ones ending with "name" and "doc" (ignore "default", etc.) get set intersection to remove the ones that do not have both fields present get a list of all field names from higher levels of hierarchy: fields:1:type:1:fields is one of parents of fields:1:type:1:fields:0:type:1:fields (the condition is that they have the same start and they end with "fields")
Question: I try to get ttm values of the income statement for ticker symbol AAPL by using from yahoo_fin import stock_info as si import yfinance as yf import pandas as pd import matplotlib.pyplot as plt import pandas_datareader pd.set_option('display.max_columns', None) income_statement = si.get_income_statement("aapl") income_statement  but the result doesn't show the ttm values, only the yearly values are shown  On yahoo finance, we can also see the ttm values:  Anyone who can help?

Answer: The reason you don't get the exact same table you see on the website is because of the way yahoo_fin gets data from Yahoo. Rather than getting them from the table you see, they get them from json data that Yahoo provides. In this data, there are both quarterly and yearly income statements. When Yahoo renders the table on their website, they most likely use the yearly data for the yearly columns and then sum the last 4 quarterly results to get the TTM column (as TTM results are nothing else than the sum of the last 4 quarterly results). If you want to get the TTM data, the best approach would be to do it the same way I assume Yahoo does. Get the quarterly data using yahoo_fin and then sum the quarters to calculate the TTM results. You can do this by setting the optional yearly parameter to False: quarterly_income_statement = si.get_income_statement("aapl", yearly=False)  You can check out their method _parse_json to better understand how they get and parse data from Yahoo. (Assuming you have some knowledge of requests and json.) Summing the data To get the sum of the quarters you can for example do this: quarterly_income_statement = si.get_income_statement("aapl", yearly=False) ttm = quarterly_income_statement.sum(axis=1)  This will give you a new Dataframe ttm with the same data fields with values being TTM (You can test it and see if it matches the numbers on the website).
Question: When I import SciPy or a library dependent on it, I receive the following warning message: UserWarning: A NumPy version >=1.16.5 and <1.23.0 is required for this version of SciPy (detected version 1.23.1  It's true that I am running NumPy version 1.23.1, however this message is a mystery to me since I am running SciPy version 1.7.3, which, according to SciPy's documentation, is compatible with NumPy <1.24.0. Anyone having this problem or know how to resolve it? I am using Conda as an environment manager, and all my packages are up to date as far as I know.  python: 3.9.12 numpy: 1.23.1 scipy: 1.7.3  Thanks in advance if anyone has any clues !

Answer: According to the setup.py file of the scipy 1.7.3, numpy is indeed <1.23.0. As @Libra said, the docs must be incorrect. You can:  Ignore this warning Use scipy 1.8 Use numpy < 1.23.0  Edit: This is now fixed in the dev docs of scipy https://scipy.github.io/devdocs/dev/toolchain.html