Myashka/SO-Python_QA-filtered-2023-tanh_score-after_2023_02 huggingface dataset:
Title: Displaying total progress bar when spliting array in portions with multiprocesssing
Tags: python,multiprocessing,tqdm

Question: I have a code where I am computing the average of an array whislt displaying a progress bar.
Here is my code:
import numpy as np
from tqdm import tqdm
import multiprocessing as mtp

def split_list(some_list, size):
  for i in range(0, len(some_list), size):
      yield some_list[i:i + size]  


def foo(x):
    return np.square(x)

def main():
    a = np.arange(1000)
    av = 0
    with mtp.Pool(4) as pool:
        for l in split_list(a, 33):
            output = list(tqdm(pool.imap(foo, l), total = len(a)))
            av += np.sum(output)
        av /= len(a)
        
    return av

This is showing several progress bars. I would like to only have one with the overall progress.
How can I do this?
EDIT:
I have something that sudo-works
def main():
    a = [np.random.rand(10, 3, 3) for _ in range(1000)]
    av = 0
    
    with mtp.Pool(4) as pool:
      with tqdm.tqdm(total = len(a)) as p_bar:
        for l in split_list(a, 33):
            output = list(pool.imap(foo, l))
            av += np.sum(output)
            p_bar.update(len(l))

    av /= len(a)
    return av

But it will only update in chuncks not as the results are available.

Answer: I think it is simply the place you have placed the "tqdm" class. Each time the loop "for l in split_list(a, 33)" enters a new value, a new progress bar is generated.
It also can happen that if the "multiprocessing" module function you are using prints messages, either through "print" or "LOG", this messages will interrupt the progress bar, and a new bar will be generated under the message with the progress of the previous bars accumulated.
If you only want one single progress bar, you should place it in the "for" loop statement and check for printed messages.
IMPORTANT: I have never worked with multiprocessing, I'm just talking about the "tqdm" module.
Title: Why Snakemake prefers calling script using script directive instead of calling from shell?
Tags: python,bioinformatics,snakemake

Question: Snakemake rules in standardized workflows run Python scripts using the script directive, such as this template rule:
rule XXXXX:
    input:
        ...,
    output:
        ....,
    params:
        ...,
    conda:
        "../envs/python.yaml"
    script:
        "../scripts/XXXX.py"

Then in the script, it is possible to use snakemake object. However, the script is then tightly coupled with that rule, which seems a big disadvantage.
Why is this approach preferred to the approach using shell that calls the script, such as in this rule?
rule XXXXX:
    input:
        ...,
    output:
        ....,
    params:
        absolute_script_path = ..., # get
        argument1 = ..., 
    conda:
        "../envs/python.yaml"
    shell:
        "python {params.absolute_script_path} {input} {params.argument1} > {output}"

In this approach, python script is decoupled from the Snakemake rule. Also it looks more cohesive, as called arguments are clear from the rule, not hidden in the script.
I am only starting with writing Snakemake workflows, so I am just a beginner. I do not understand why the first approach is preferred (or used in standardized Snakemake workflows) to the second approach? Am I missing something? Are there some problems with the second approach?
Thank you very much for answers!

Answer: The script approach is a bit more flexible in terms of the objects that the script can access via the params and other directives.
If you follow the shell approach you might find it cumbersome to (re) define the argparse or other approaches to properly take account of the arguments passed via shell. It's going to be mostly boilerplate, but can get somewhat tedious.
The notebook directive might be useful in scenarios that require interactive reproduction/development.
All in all, there are no hard rules, and for a given workflow one approach might be more suitable/convenient than other approaches.
Title: Duplicates, and strange values after merging dataframes
Tags: python,python-3.x,pandas,pandas-merge

Question: I am using Python 3.
I have a master dataframe " df " with the columns as shown (with 3 rows of sample data):
UNITID  CIPCODE AWLEVEL CTOTALT
100654  1.0999  5        9
100654  1.1001  5        10
100654  1.1001  7        6

I have a dataframe called " uni_names " as shown (with 3 rows of sample data):
UNITID    institution_name
100654    Alabama A & M University
100663    University of Alabama at Birmingham
100690    Amridge University

I have a dataframe called " cipcodes " as shown (with 3 rows of sample data):
cipcode_value   program_name
01.0000         Agriculture, General
01.0101         Agricultural Business and Management, General
01.0102         Agribusiness/Agricultural Business Operations

I have a dataframe called " awlevel " as shown (with 3 rows of sample data):
code    type    
3       Associate's degree
5       Bachelor's degree
7       Master's degree


What I want is an output dataframe with column names as such
institution_name    program_name    type    CTOTALT


My code below is giving duplicates and weird additional values:
import pandas as pd

# Read the master dataframe from a CSV file

df = pd.read_csv('master_data.csv')

# Read the uni_names dataframe from a CSV file

uni_names = pd.read_csv('uni_names.csv')

# Read the cipcodes dataframe from a CSV file

cipcodes = pd.read_csv('cipcodes.csv')

# Read the awlevel dataframe from a CSV file

awlevel = pd.read_csv('awlevel.csv')

# Merge df with uni_names based on UNITID

merged_df = df.merge(uni_names, on='UNITID')

# Merge merged_df with cipcodes based on CIPCODE

merged_df = merged_df.merge(cipcodes, left_on='CIPCODE', right_on='cipcode_value')

# Merge merged_df with awlevel based on AWLEVEL

merged_df = merged_df.merge(awlevel, left_on='AWLEVEL', right_on='code')

# Select the desired columns and assign new column names

output_df = merged_df[['institution_name', 'program_name', 'type', 'CTOTALT']]

output_df.columns = ['institution_name', 'program_name', 'type', 'CTOTALT']

# Print the output dataframe

print(output_df)

Answer: actually there were two colleges by the same name! So a bit of data exploration
revealed that there were no duplicates
Title: Memory leak after calling Keras model.fit()
Tags: python,tensorflow,keras,memory-leaks

Question: I wrote my custom Keras layer called Material and tried using it in my model:
import tensorflow as tf
from tensorflow import keras
import numpy as np

class Material(keras.layers.Layer):
    def __init__(self):
        super().__init__()
        self.table = tf.Variable(
            initial_value=np.ones(shape=(6, 8, 8), dtype="float32"),
            trainable=True,
        )
        self.k = tf.Variable(initial_value=8., trainable=True, dtype='float32')
    
    def call(self, inp):
        material_white = tf.reduce_sum(inp[..., 0] * self.table, axis=(-1, -2, -3))
        material_black = tf.reduce_sum(inp[..., 1] * self.table, axis=(-1, -2, -3))
        material_white = tf.maximum(material_white, .01)
        material_black = tf.maximum(material_black, .01)
        return tf.math.log(material_white / material_black) * self.k

def get_model() -> keras.Model:
    inp = keras.Input((6, 8, 8, 2),)
    material = Material()
    out = keras.activations.sigmoid(material(inp))
    return keras.Model(inputs=inp, outputs=out)

model = get_model()
model.compile(optimizer=keras.optimizers.Adam(learning_rate=.01), loss='mean_squared_error')

board_tables = np.random.sample((100000, 6, 8, 8, 2),)
outcome = np.random.sample((100000,),)

while 1:
    model.fit(board_tables, outcome, batch_size=200, epochs=2)

For some reason it turns out to use more and more RAM after each fit iteration (I mean function call, not each epoch). At around 5 GB the memory usage growth slows down but still continues. And the problem is present both on CPU and GPU. Could anyone explain what is going on? Is there something wrong with my custom layer?
Thanks for any suggestions.

Answer: I used tf.data.Dataset created from custom generator to pass data into my model. For some reason the memory leak is gone now. But i still don't understand what's wrong with regular np arrays.
Title: Tkinter - memory leak arising from "Frame.pack"
Tags: python,tkinter,memory-leaks

Question: I have a TKinter application (running on MacOS, M1 laptop) that allows you to select a video or livestream, and play it in an adjacent frame.
When it's a livestream, there is no need for "video progress" slider, so I hide it.
Here is the critical code:
class DetectionVideoPlayer:
   ...
   def __init__(self, ): 
       self.progress_status_bar = tk.Frame(...)

    def _reset_ui_widgets(self, is_live: bool = False):
        self._progress_status_bar.pack_forget()
        if not is_live:
            self._progress_status_bar.pack(fill=tk.X, expand=True, side=tk.BOTTOM)

Now, when transitioning from a livestream (is_live=True), to a video (is_live=False), My entire application freezes and I get this rapid memory leak which (if I don't kill the process) will actually freeze the computer.  Commenting out self._progress_status_bar.pack(...) gets rid of the leak.  The line self._progress_status_bar.pack(...) is actually executed fairly quickly and the interpreter passes by it, the application freezes thereafter (I presume on the next draw cycle).
I'm trying to understand why this is happening.  Does anyone here understand why calling Frame.pack can freeze the application, and why it can cause a (very rapid, like 1GB every 5s or so) memory leak?

Answer: Got it (thank you GPT-4 for the suggestion).
The problem arose from the children of self.progress_status_bar (who had been packed in __init__).  When I change the code in _reset_ui_widgets to also call .pack_forget() on the children, and then repack them when appropriate, the freezing and memory leak problems go away.
Title: Descriptor's __set__ not invoked
Tags: python,python-decorators,inner-classes,python-descriptors

Question: I have a innerclass decorator/descriptor that is supposed to pass the outer instance to the inner callable as the first argument:
from functools import partial

class innerclass:

  def __init__(self, cls):
    self.cls = cls

  def __get__(self, obj, obj_type=None):
    if obj is None:
      return self.cls

    return partial(self.cls, obj)

Here's a class named Outer whose .Inner is a class decorated with innerclass:
class Outer:

  def __init__(self):
    self.inner_value = self.Inner('foo')

  @innerclass
  class Inner:

    def __init__(self, outer_instance, value):
      self.outer = outer_instance
      self.value = value

    def __set__(self, outer_instance, value):
      print('Setter invoked')
      self.value = value

I expected that the setter would be invoked when I change the attribute. However, that is not the case:
foo = Outer()
print(type(foo.inner_value))  # <class '__main__.Outer.Inner'>

foo.inner_value = 42
print(type(foo.inner_value))  # <class 'int'>

Why is that and how can I fix it?

Answer: inner_value is inside of your instance's __dict__. Descriptors apply at the class level. If you assign to foo.bar, then Python looks for a settable descriptor on the type of foo and its parents, not on the dictionary of foo itself. You cannot have descriptors that apply on a per-instance basis; Python just isn't built that way. You have to place the descriptor on a class and then apply it to instances of that class.
Title: Why is pip reporting dependencies conflicts even though there are none?
Tags: python,numpy,build,pip,pip-tools

Question: pip is reporting dependencies conflicts when there are none:
The conflict is caused by:
    apache-beam[gcp] 2.39.0 depends on numpy<1.23.0 and >=1.14.3
    tensorflow-utils 0.0.18.dev1 depends on numpy==1.22.4

tensorflow-utils==0.0.18.dev1 is my own package. The boundaries in the conflict specify that numpy must be >=1.14.3 and <1.23 for apache-beam[gcp], which my numpy==1.22.4 version satisfies.
Environment details:
root@6b7ea6e22c5a:/# python3 -V
Python 3.7.10

root@6b7ea6e22c5a:/# python3 -m pip -V
pip 21.2.2 from /opt/conda/lib/python3.7/site-packages/pip (python 3.7)

Answer: It could be cached dependency information: It's possible that pip has cached outdated or conflicting dependency information. You can try clearing the pip cache pip cache purge
Title: Why doesn't pandas.sum() work across both axes when using axis=None parameter?
Tags: python,pandas,sum

Question: Could anyone explain why pandas doesn't sum across both axes with parameter axis=None. As it said in API reference:

pandas.DataFrame.sum
DataFrame.sum(axis=None, skipna=True, numeric_only=False, min_count=0, **kwargs)
This is equivalent to the method numpy.sum
Parameters: axis: {index (0), columns (1)}
Axis for the function to be applied on. For Series this parameter is unused and defaults to 0.
For DataFrames, specifying axis=None will apply the aggregation across both axes.

But when I use parameter axis=None it works the same as axis=0
import pandas as pd
df = pd.DataFrame({'a':[1,2,3], 'b':[4,6,8]})
df

Output:
    a   b
0   1   4
1   2   6
2   3   8

df.sum(axis=None)

Output:
a     6
b    18
dtype: int64

The same as:
df.sum(axis=0)

Output:
a     6
b    18
dtype: int64

Shouldn't it work as numpy.sum() works?
import numpy as np
df.to_numpy().sum()

Output:
24

Answer: The axis=None parameter in the pandas.sum() function does not work across both axes when there are non-numeric values present in the DataFrame or Series. It only works for numeric data.
Title: Python project not importing a local module
Tags: python,flask,apache2

Question: I have a python project using Flask and I have a form I have setup in a module I called forms. I test my app in Windows and it works just fine. I then update my project on Debian where I use Apache2 to run the project. There I get the error: ModuleNotFoundError: No module named 'forms'
My project is organized like so:
flaskapp.wsgi
flask_app
    __init__.py
    forms.py

And __init__.py starts with:
from flask import Flask, redirect, url_for, request, render_template, send_from_directory, abort
from requests import Request, Session
from forms import OrderForm

I checked the sys.path with:
import sys
print(sys.path)

when using the python console and got:
['', '/usr/lib/python37.zip', '/usr/lib/python3.7', '/usr/lib/python3.7/lib-dynload', '/usr/local/lib/python3.7/dist-packages', '/usr/lib/python3/dist-packages']

I'm not sure what I need to do to get it to use forms as a module

Answer: Try from .forms import OrderForm.
This is referencing to the same folder. Otherwise flask_app.forms should work too.
Title: Automatically create database in python application
Tags: python,mysql,database,mariadb

Question: I am relatively new to application development, but I'm working on a personal project and I would like it to automatically deploy a mariadb/mysql db on first install/through an option in the application. Now, I understand how to create a db, after the mariadb server has been set up, and I've got that part implemented and working. But what I would like to do is not have to install mariadb, configure the server, etc, and have the application handle that automatically. I feel like it must be possible, but I haven't been able to find an answer on how to implement it.

Answer: You can't have your application automatically install MariaDB & configure it on the machine you're running it on. And even if it could, then you wouldn't want to do that.
If you were to automatically install and configure the DB, then whenever you run your program for the first time on a new machine, it could take a really long time to install the DB. If you do want to automatically install & configure, then you're better off writing (or finding online) a bash script to do it. Then, you can just run the script separately and don't have to worry about any unexpected side effects.
Also, most of the time in production your DB isn't even on the same machine as your web app, especially if you're running it on Docker or some other form of containerization. The point of this is to take the strain off of the web app and let another isolated machine handle all the DB stuff. You could also use a service to handle your DB for you, so you don't have to install or configure anything, just provide your web app the URL, username, and password of your DB. This is likely your best bet if you don't want to do any DB configuration. I won't spend any time here listing services that can do it for you, but you can find hundred by just doing a Google search for "MariaDB hosting".
Title: Why doesn't psutil accurately represent available memory after deleting a class instance in Python?
Tags: python,memory-management,psutil,del

Question: I'm currently struggling to find out how much memory I actually have left on my device after I have deleted an instance of a class. I have tried to use the library psutil but it doesn't correctly show the memory I actually have available. In particular:
import psutil

    x = torch.randint(0,255,size=(300,3,200,200),dtype=torch.float32)
    conv = Modules.Conv2D(3,3,2,10,save_flag=False)
 print("Used memory:", psutil.virtual_memory().percent,"% Free memory:", round(psutil.virtual_memory().available * 100 /
          psutil.virtual_memory().total),"%")
#Used memory: 74.0% Free memory: 26.0 %


conv is an object of my custom class and It has different attributes that take up a lot of memory(as you can see from psutil). Now If I try the following:
import gc

del conv
gc.collect()
 print("Used memory:", psutil.virtual_memory().percent,"% Free memory:", round(psutil.virtual_memory().available * 100 /
          psutil.virtual_memory().total),"%")
#Used memory: 74.0% Free memory: 26.0 %

Even if I delete the instance and I explicitly call python's garbage collector, the memory doesn't seem to be freed up. Although if I check with Windows' resource manager the memory is actually emptied:
#Before executing the program --> In use: 6025 MB
exec_program()
#Program is executing and conv instance is allocated --> In use: 6899 Mb
# gc.collect is called --> In use: 6058 Mb
end_of_program()

So the problem is that psutil doesn't represent correctly this situation. My guess is that although the memory is freed up by gc.collect the virtual memory address is still marked as used even if it is actually empty, hence the misrepresentation. Is there any other way/library that can do correctly what I'm trying to achieve?
Thanks in advance!

Answer: Oops, memory allocation and deallocation is often a grey zone, where few things are guaranteed, either at Python level, or even at OS one. Specifically you cannot know whether deleting something at Python level will put the freed memory into a (process) local free list or will return it to the OS.
That local free list exists because when a program does a lot of memory allocation/deallocation, it is far more efficient to keep the memory local than to return it back to the OS to later ask for it again. Said differently, psutil does its job and tells you how much memory is given to a specific process. But it does not know (hence cannot tell you) how the memory is used by the process...
Title: Odoo dynamically select views to render
Tags: python,odoo,odoo-16

Question: I have create 4 form view from odoo user interface. Now I want to render view conditionally based on my model data like state.
How can I achieve this ? or
Where I can override a method to render view conditionally?
Update
According to the comment overrides the get_view method  works but to refresh the form page. When I refresh the form page my custom view applies to the form page. How Can I Fix this cache issue issue?
def get_view(
        self, view_id=None, view_type='form', **options
):
    # Call the super method to get the original view definition
    result = super().get_view(
        view_id=view_id, view_type=view_type, **options
    )
    if view_type == 'form' and 'params' in self._context and 'id' in \
            self._context['params']:
        # get record ID from _context when access Form page
        record = self.browse(self._context['params']['id'])
        view = self.env['ir.ui.view'].search(
            [('name', '=', f'My Custom View')], limit=1
        )
        result.update(
            {
                'id': view.id,
                'arch': view.arch_db,
                'model': 'custom_module.model_name',
                'models': {
                    'custom_module.model_name': model_fields,
                    f'related_model': related_model_fields
                },
            }
        )
    return result

Answer: An easy solution would be to trigger a function clicking on a button.
This function could decide which xml_id view to use and then return an action to this view.
The clean solution would be to do a js class and to apply it in the view.
The JS class could be responsible to chose its own OWL template depending on the data it gets from the arch.
Title: Long or background callbacks in django-plotly-dash
Tags: python,python-3.x,django,callback,plotly-dash

Question: I have been using the django-plotly-dash package lately and was wondering if I could implement a progress bar for a very long-running task of my app.
Below I have the demo-version of the progress bar I would like to implement. Is it even possible in the django-plotly-dash package? If not does anyone have any suggestions how I should deal with that?
I'm very thankful for any replies helping since I'm kinda stuck...
import time
import os

import dash
from dash import DiskcacheManager, CeleryManager, Input, Output, html

if 'REDIS_URL' in os.environ:
    # Use Redis & Celery if REDIS_URL set as an env variable
    from celery import Celery
    celery_app = Celery(__name__, broker=os.environ['REDIS_URL'], backend=os.environ['REDIS_URL'])
    background_callback_manager = CeleryManager(celery_app)

else:
    # Diskcache for non-production apps when developing locally
    import diskcache
    cache = diskcache.Cache("./cache")
    background_callback_manager = DiskcacheManager(cache)

app = dash.Dash(__name__, background_callback_manager=background_callback_manager)

app.layout = html.Div(
    [
        html.Div(
            [
                html.P(id="paragraph_id", children=["Button not clicked"]),
                html.Progress(id="progress_bar", value="0"),
            ]
        ),
        html.Button(id="button_id", children="Run Job!"),
        html.Button(id="cancel_button_id", children="Cancel Running Job!"),
    ]
)

@dash.callback(
    output=Output("paragraph_id", "children"),
    inputs=Input("button_id", "n_clicks"),
    background=True,
    running=[
        (Output("button_id", "disabled"), True, False),
        (Output("cancel_button_id", "disabled"), False, True),
        (
            Output("paragraph_id", "style"),
            {"visibility": "hidden"},
            {"visibility": "visible"},
        ),
        (
            Output("progress_bar", "style"),
            {"visibility": "visible"},
            {"visibility": "hidden"},
        ),
    ],
    cancel=Input("cancel_button_id", "n_clicks"),
    progress=[Output("progress_bar", "value"), Output("progress_bar", "max")],
    prevent_initial_call=True
)
def update_progress(set_progress, n_clicks):
    total = 5
    for i in range(total + 1):
        set_progress((str(i), str(total)))
        time.sleep(1)

    return f"Clicked {n_clicks} times"


if __name__ == "__main__":
    app.run_server(debug=True)

Answer: I would be interested in exactly the same question: how to use long/background callbacks with django-plotly-dash?
Title: import langchain => Error : TypeError: issubclass() arg 1 must be a class
Tags: python,nlp,data-science,chatbot,langchain

Question: I want to use langchain for my project.
so I installed it using following command : pip install langchain
but While importing "langchain" I am facing following Error:
File /usr/lib/python3.8/typing.py:774, in _GenericAlias.__subclasscheck__(self, cls)
    772 if self._special:
    773     if not isinstance(cls, _GenericAlias):
--> 774         return issubclass(cls, self.__origin__)
    775     if cls._special:
    776         return issubclass(cls.__origin__, self.__origin__)

TypeError: issubclass() arg 1 must be a class

Any one who can solve this error ?

Answer: typing-inspect==0.8.0
typing_extensions==4.5.0
Title: import langchain => Error : TypeError: issubclass() arg 1 must be a class
Tags: python,nlp,data-science,chatbot,langchain

Question: I want to use langchain for my project.
so I installed it using following command : pip install langchain
but While importing "langchain" I am facing following Error:
File /usr/lib/python3.8/typing.py:774, in _GenericAlias.__subclasscheck__(self, cls)
    772 if self._special:
    773     if not isinstance(cls, _GenericAlias):
--> 774         return issubclass(cls, self.__origin__)
    775     if cls._special:
    776         return issubclass(cls.__origin__, self.__origin__)

TypeError: issubclass() arg 1 must be a class

Any one who can solve this error ?

Answer: Install langchain version 0.0.133. After this version, the error occurs.
Title: import langchain => Error : TypeError: issubclass() arg 1 must be a class
Tags: python,nlp,data-science,chatbot,langchain

Question: I want to use langchain for my project.
so I installed it using following command : pip install langchain
but While importing "langchain" I am facing following Error:
File /usr/lib/python3.8/typing.py:774, in _GenericAlias.__subclasscheck__(self, cls)
    772 if self._special:
    773     if not isinstance(cls, _GenericAlias):
--> 774         return issubclass(cls, self.__origin__)
    775     if cls._special:
    776         return issubclass(cls.__origin__, self.__origin__)

TypeError: issubclass() arg 1 must be a class

Any one who can solve this error ?

Answer: Fixed in Pydantic 1.10.8.
For poetry, use pydantic = "^1.10.8" in your pyproject.toml file.
Title: AsyncIO appears to be slower than multithreaded even if tasks are only bound to I/O
Tags: python,multithreading,python-asyncio

Question: I am creating 8000 text files and writing 100 bytes of data to each file in both asyncio and multithread version. I was expecting asyncio version of the code to be completed before multithread version of the code , as all tasks related to I/O bound and I have understood that asyncio will perform better but the situation is other way around. Asyncio is not only perfroming slow but it is also taking more space than the thread version of the code.
I must be doing something wrong, I have tried it to find where I am lacking in my code but didn't found as such.
Async IO Version
import time

import aiofiles
import asyncio
from  timeit import timeit
DATA="A"*100
async def write_file(filename_plus_path):
    async with aiofiles.open(filename_plus_path,"w") as f:
        await f.write(DATA)

async  def main():
    coro = [asyncio.create_task(write_file(f"./file/text{i}.txt")) for i in range(8000)]
    await coro[-1]
st = time.time()
print("Started at:",time.strftime("%X"))
asyncio.run(main())
print("Finished at:",time.strftime("%X"))
res = time.time() - st
final_res = res * 1000
print('Execution time:', final_res, 'milliseconds')

Output:
Started at: 16:15:39
Finished at: 16:15:44
Execution time: 4243.027448654175 milliseconds

Multithread Version
import time

import threading
DATA="A"*100
def write_file(filename_plus_path):
   with open(filename_plus_path,"w") as f:
        f.write(DATA)


print("Started at:",time.strftime("%X"))
st = time.time()
thrs=[threading.Thread(target=write_file,args=(f"./file/text{i}.txt",)) for i in range(8000)]
for th in thrs:
    th.start()
for th in thrs:
    th.join()
# print("Finished at:",time.strftime("%X"))
res = time.time() - st
final_res = res * 1000
print('Execution time:', final_res, 'milliseconds')
print("Finished at:",time.strftime("%X"))

output:
Started at: 16:16:57
Execution time: 3494.1318035125732 milliseconds
Finished at: 16:17:01

Answer: File access in Python aiofiles is not proper "single-thread async": it just use multi-threading under the hood - by default, the a ThreadPoolExecutor as provided by default by the async module.
A "real" asyncio file thing should use selectors -and for LInux, there are some libraries based on the new IO_Uring async file access.
That said, the CPU bound overhead of writting 100 bytes  overwhelms any gain one would get of telling the O.S. to store 100 bytes in a file - keep in mind that the least amount of bytes that will touch the "disk surface" will be 4096 in most scenarios.
Anyway, your answer oficcialy is: aiofiles async file implementation uses multi-threading under the hood, with extra limitations due to the use of a default TheadPoolExecutor (I think the number of worker threads equal the number of CPU-Cores)  -
You will likely get different results, no guarantees for what side of it :-), using a io_uring based async file writing library (or a selector based one),and meaningful quantities of bytes to each file - say, starting from 1MB.
Title: Retrieve the MAC address of a device through a Python program while connecting two computers via cable?
Tags: python,configuration,wifi

Question: How can we use python code to connect two computers via cable and get mac address of one device in a small window?
Have to configure multiple devices for wifi connectivity in my domain. finding it difficult to type mac address of each device coming for configuration.

Answer: Your wifi access point is probably a router with DHCP server. And your device is probably DHCP enabled. Your device is asking for an IP address from your router at power-up.
Look at your router report too get the IP and MAC.
Title: FastAPI SQLAlchemy cannot convert dictionary update sequence element #0 to a sequence
Tags: python,sqlalchemy,fastapi

Question: I'm trying to return list of operations and getting error
@router.get("/")
async def get_specific_operations(operation_type: str, session: AsyncSession = Depends(get_async_session)):
    query = select(operation).where(operation.c.type == operation_type)
    result = await session.execute(query)
    return result.all()

Error:
ValueError: [TypeError('cannot convert dictionary update sequence element #0 to a sequence'), TypeError('vars() argument must have __dict__ attribute')]

Answer: try instead of result.all() to do this result.scalars().all()
Title: Python - Count all combinations of K numbers from 1-N whose sum is equal to N
Tags: python,algorithm

Question: How do i count all combinations of k numbers from 1-n whose sum is equal to n? Like for n = 10, k = 3, we have (1, 2, 7), (1, 3, 6), (1, 4, 5), (2, 3, 5)
I've tried using itertools.combination but it grows really fast for big numbers

Answer: Let's introduce a function: f(n,k,s)=number of combinations of k numbers from 1 to n, having s as their sum.
To solve the task we need to calculate f(n,k,n).
The function may be calculated recursively. All combinations can be split into two groups: with and without the max number. That gives us f(n,k,s)=f(n-1,k-1,s-n)+f(n-1,k,s). The recursion may stop in the following cases:

n<k -> 0 (we don't have enough numbers)
k=1, s>n -> 0 (every number is too small)
k=1, s<1 -> 0 (every number is too small)
k=1, 1<=s<=n -> 1 (there is only one suitable number)
s<0 -> 0

There are N^2*k possible combinations of arguments, so if we cache the already calculated values, we will be within O(N^3).
Title: How to do partial matching on searches with PostgreSQL in wagtail?
Tags: python-3.x,django,wagtail,wagtail-search

Question: How to do partial matching on searches with PostgreSQL as search backend?
I'm running wagtail 5.0 and I see the following statements in the documentation

The partial_match option has been deprecated. To index a field for partial matching, use AutocompleteField instead.

However in the autocomplete documentation it states

This method should only be used for real-time autocomplete and actual search requests should always use the search() method.

What is the issue with using the autocomplete method vs search? Functionally the autocomplete method gives me what i want.
>>> UtvalgPostPage.objects.all().live().search("alt")
<SearchResults [<UtvalgPostPage: Demo på alt vi har>]>
>>> UtvalgPostPage.objects.all().live().search("al")
<SearchResults []>
>>> UtvalgPostPage.objects.all().live().autocomplete("al")
<SearchResults [<UtvalgPostPage: Demo på alt vi har>]>

Answer: As I understand it, the second note is simply providing guidance on implementing a search interface that behaves in the conventional way: if you're running a search in the classic "submit search term, receive results" way, then you wouldn't want a search for "cat" to return results about caterpillars, but on a search interface that suggests results in real-time as you type, it's legitimate for it to return those results immediately after typing "cat".
However, if the autocomplete method gives you the result you want, there's no problem with using that. The documentation shouldn't really be giving instructions in such absolute terms - I'll open a pull request to update that.
Title: Django 4.1.9 requires_system_checks issue with manage.py - is this a bug or not
Tags: python,django

Question: Django 4.1.9 requires_system_checks issue with manage.py - is this a bug or not
we are upgrading our wagtail 4.2.2 app to from django 3.1.9 to django 4.1.9
getting the error
TypeError: requires_system_checks must be a list or tuple.

when running 

python manage.py runserver_plus 0.0.0.0:8020 --keep-meta-shutdown


Is this a bug and / or can i be worked around
Could I monkey patch BaseCommand.init() to set requires_system_checks = []

Answer: Resolved issue by upgrading dependancy django-extensions to latest version 3.2.1
Title: percent-encoded %2F fail request
Tags: python,python-requests

Question: I have a dictionary that I pass as a parameter in request:
paylod = {
    'latitude': 42.406747, 
    'longitude': 12.154226, 
    'timezone': 'Europe%2FBerlin',  # encoding problem
    'time_format': 'iso8601', 
    'temperature_unit': 'celsius', 
    'wind_speed_unit': 'ms', 
    'precipitation_unit': 'mm', 
    'start_date': '2023-05-20', 
    'end_date': '2023-05-23', 
    'models': ['best_match', 'era5', 'era5_land'], 
    'hourly': ['temperature_2m', 'relativehumidity_2m', 'dewpoint_2m', 'apparent_temperature', 'pressure_msl', 'surface_pressure', 'precipitation', 'rain', 'snowfall', 'weathercode', 'cloudcover', 'cloudcover_low', 'cloudcover_mid', 'cloudcover_high', 'shortwave_radiation', 'direct_radiation', 'diffuse_radiation', 'direct_normal_irradiance', 'windspeed_10m', 'windspeed_100m', 'winddirection_10m', 'winddirection_100m', 'windgusts_10m', 'et0_fao_evapotranspiration', 'vapor_pressure_deficit', 'soil_temperature_0_to_7cm', 'soil_temperature_7_to_28cm', 'soil_temperature_28_to_100cm', 'soil_temperature_100_to_255cm', 'soil_moisture_0_to_7cm', 'soil_moisture_7_to_28cm', 'soil_moisture_28_to_100cm', 'soil_moisture_100_to_255cm']
    }

or
import request
api_url = "https://archive-api.open-meteo.com/v1/archive"
response = requests.get(
        url= api_url,
        params=payload
    )

The problem concerns the encoding of the Europe%2FBerlin parameter (i.e., timezone). In fact, in the final url, which causes the request to fail, the parameter Europe%2FBerlin appears as Europe%252FBerlin
// https://archive-api.open-meteo.com/v1/archive?latitude=42.406747&longitude=12.154226&timezone=Europe%252FBerlin&time_format=iso8601&temperature_unit=celsius&wind_speed_unit=ms&precipitation_unit=mm&start_date=2023-05-20&end_date=2023-05-23&models=best_match&models=era5&models=era5_land&hourly=temperature_2m&hourly=relativehumidity_2m&hourly=dewpoint_2m&hourly=apparent_temperature&hourly=pressure_msl&hourly=surface_pressure&hourly=precipitation&hourly=rain&hourly=snowfall&hourly=weathercode&hourly=cloudcover&hourly=cloudcover_low&hourly=cloudcover_mid&hourly=cloudcover_high&hourly=shortwave_radiation&hourly=direct_radiation&hourly=diffuse_radiation&hourly=direct_normal_irradiance&hourly=windspeed_10m&hourly=windspeed_100m&hourly=winddirection_10m&hourly=winddirection_100m&hourly=windgusts_10m&hourly=et0_fao_evapotranspiration&hourly=vapor_pressure_deficit&hourly=soil_temperature_0_to_7cm&hourly=soil_temperature_7_to_28cm&hourly=soil_temperature_28_to_100cm&hourly=soil_temperature_100_to_255cm&hourly=soil_moisture_0_to_7cm&hourly=soil_moisture_7_to_28cm&hourly=soil_moisture_28_to_100cm&hourly=soil_moisture_100_to_255cm

{
  "reason": "Invalid timezone",
  "error": true
}

Answer: The problem is that request.get convert "%" into "%25" in URL encoding.
Change 'timezone': 'Europe%2FBerlin' into 'timezone': 'Europe/Berlin' should work.
Title: Python Poetry fails to add openai-whisper due to triton installation error
Tags: python-3.x,python-poetry,openai-whisper

Question: so im trying to use openai-whisper. i'm using poetry as my env and dependecy manager. but keep getting errors when trying to download it. the error i get is, Installing triton (2.0.0): Failed
i tried the typical poetry add and this is the error
poetry add openai-whisper 
Using version ^20230314 for openai-whisper

  • Installing triton (2.0.0): Failed

  RuntimeError

  Unable to find installation candidates for triton (2.0.0)

  at /usr/local/Cellar/poetry/1.4.2/libexec/lib/python3.11/site-packages/poetry/installation/chooser.py:109 in choose_for
      105│ 
      106│             links.append(link)
      107│ 
      108│         if not links:
    → 109│             raise RuntimeError(f"Unable to find installation candidates for {package}")
      110│ 
      111│         # Get the best link
      112│         chosen = max(links, key=lambda link: self._sort_key(package, link))

Answer: Most likely triton isnt available for combination of your platform + Python version.
When I stumble upon similar issue, I try to install the package manually via pip (pip install triton==2.0.0) and that usually gives me more detailed error message what went wrong.
If this wont help, please update the question with following:

what's the version of Python in your pyproject.toml?
what's your platform?
Title: Why is asyncio.Future.done() not set to True when the task is done?
Tags: python,asynchronous,python-asyncio

Question: In this example code a asyncio.Future is created and run. However the state is not set to done once it is complete.
import asyncio
from concurrent.futures.thread import ThreadPoolExecutor
from time import sleep

_executor = ThreadPoolExecutor(max_workers=32)

def test():
    print('starting test')
    sleep(5)
    print('ending test')

async def main():
    loop = asyncio.get_running_loop()
    result = loop.run_in_executor(_executor, test)

    sleep(3)
    print('sleep 1 complete')
    print(f'{result.done()=}')

    sleep(3)
    print('sleep 2 complete')
    print(f'{result.done()=}')

    print('await result')
    await result
    print(f'{result.done()=}')

asyncio.run(main())

This results in:
starting test
sleep 1 complete
result.done()=False
ending test
sleep 2 complete
result.done()=False
await result
result.done()=True

Why is result.done() set to False after the second sleep?

Answer: You are not yielding the control to the asyncio loop in the first steps in your snippet: the asyncio loop never gets a chance to check the Future status and set whatever in it.
For one take away, make sure you understand this: no async related code can run unless you pass control to the loop with one of await, async with or async for
There is no, "with enough sleep, eventually the asyncio will kick in and do its thing". No. It is single threaded code, and one of the benefits of using it is giving the user full control of where the task changes take place. And in your code, they take place just in the second-to-last line with await result.
Yes, in this case, the task itself is dispatched to another thread in the executor, and it runs there - but it is the asyncio loop which checks for the task completion, set the attributes in the Future, and so on.
Title: Python file status when open() is not assigned to any variable
Tags: python

Question: In Python it is often advised to either use a with statement or close an opened file when we are done with it to clean up gracefully. I understand that when using with the __enter__() and __exit__() methods are called. However, what if I don't create a variable from the data stream? Something like this:
import json


def read_json(file):
    return json.load(open(file))

Would this close the file automatically? Also, is there a way to test this? Thanks.

Answer: Will it close automatically?
Yes.
Upon exiting scope the ref count decrements and the file descriptor closes.
(But don't try to get away with this in Jython!)

How to verify?
Consult os.listdir('/proc/self/fd/')
Title: How can I run multiple ngrok sessions using pyngrok and a free account?
Tags: python,ngrok,pyngrok

Question: pyngrok multiple sessions setup
Hi, i'm trying to run 2 python files with pyngrok, but i receive next error:
The ngrok process errored on start: authentication failed: Your account is limited to 1 simultaneous ngrok agent session.
You can run multiple tunnels on a single agent session using a configuration file.
To learn more, see https://ngrok.com/docs/ngrok-agent/config/

i tried to change ngrok.yml file:
version: "2"
authtoken: *********
tunnels:
  first:
    addr: 8443
    proto: http    
  second:
    addr: 8444
    proto: http

And my .py files
1 file:
public_url = ngrok.connect(8443, "http").public_url

2 file
public_url = ngrok.connect(8444, "http").public_url

How to use multiple sessions with free acc and pyngrok?

Answer: To open multiply sessions with "PyNgrok" you need:

Сhange the configuration file (which you already did)
Write in the console ngrok ->ngrok start --all
Run your web applications on these ports specified in the configuration without using PyNgrok. You just need to run localhost on the port and that's it. Next, ngrok will do everything himself.
Title: How can I run multiple ngrok sessions using pyngrok and a free account?
Tags: python,ngrok,pyngrok

Question: pyngrok multiple sessions setup
Hi, i'm trying to run 2 python files with pyngrok, but i receive next error:
The ngrok process errored on start: authentication failed: Your account is limited to 1 simultaneous ngrok agent session.
You can run multiple tunnels on a single agent session using a configuration file.
To learn more, see https://ngrok.com/docs/ngrok-agent/config/

i tried to change ngrok.yml file:
version: "2"
authtoken: *********
tunnels:
  first:
    addr: 8443
    proto: http    
  second:
    addr: 8444
    proto: http

And my .py files
1 file:
public_url = ngrok.connect(8443, "http").public_url

2 file
public_url = ngrok.connect(8444, "http").public_url

How to use multiple sessions with free acc and pyngrok?

Answer: Make sure you are using the latest version of pyngrok that uses version 3.x of the ngrok Agent. The older version uses version 2.x of the ngrok Agent which starts both an http and https tunnels for the same endpoint by default. Defining two tunnels in the config file actually starts 4 tunnels. Version 3.x only starts the https tunnel by default.
Title: Python: Type hinting a dictionary mapping types to functions
Tags: python,type-hinting,mypy

Question: I'm trying to create a dictionary mapping a tuple of input data types to a function to handle those data types. I haven't been able to figure out a way to type-hint it in a way that satisfies mypy. Before I disable static type checking on the file, I figured I'd ask...
Here's a MWE:
from typing import Callable

class BaseType: pass
class IntType(BaseType): pass
class StringType(BaseType): pass

def _compare_int_int(before: IntType, after: IntType) -> int:
    return 1

def _compare_int_str(before: IntType, after: StringType) -> int:
    return 2

def _compare_str_int(before: StringType, after: IntType) -> int:
    return 3

def _compare_str_str(before: StringType, after: StringType) -> int:
    return 4


_COMPARERS: dict[tuple[type[BaseType], type[BaseType]], Callable[[BaseType, BaseType], int]] = {
    (IntType, IntType): _compare_int_int,
    (IntType, StringType): _compare_int_str,
    (StringType, StringType): _compare_str_str,
    (StringType, IntType): _compare_str_int
}

def compare(before: BaseType, after: BaseType) -> int:
    compare_func = _COMPARERS[(type(before), type(after))]
    return compare_func(before, after)

a = IntType()
b = StringType()

print(compare(a, a))
## 1
print(compare(a, b))
## 2
print(compare(b, a))
## 3
print(compare(b, b))
## 4

It works as desired, but mypy gives errors
Dict entry 0 has incompatible type "Tuple[Type[IntType], Type[IntType]]": "Callable[[IntType, IntType], int]"; expected "Tuple[Type[BaseType], Type[BaseType]]": "Callable[[BaseType, BaseType], int]"

and likewise for the other entries.
Any tips for type-hinting this correctly?
I've
Thanks.
EDIT
I've also tried using TypeVar like so:
T = TypeVar('T', bound=BaseType)
U = TypeVar('U', bound=BaseType)

_COMPARERS: dict[tuple[type[T], type[U]], Callable[[T, U], int]] = {...}

Although I liked this idea better (it clarifies that the parameter types of the functions match the respective types of the tuples), it still gives errors:
Dict entry 0 has incompatible type "Tuple[Type[IntType], Type[IntType]]": "Callable[[IntType, IntType], int]"; expected "Tuple[Type[T?], Type[U?]]": "Callable[[T, U], int]"

The errors don't change if I leave T and U without a bound value.

Answer: I would suggest one of the following that might solve the issue:

Using TypeVar with your function is a better way to get the types.
If you're dealing with built-in types like str and int, why create a class for them? Use the original types
Title: Why does the prefix for instance variables remain the same for both the parent and child class?
Tags: python,class,inheritance,instance

Question: I have two classes, Rectangle and Square. Square is the child class to Rectangle.
My issue is when I put in a self.dict statement for either class, to check the values for their instances, both have the prefix _Rectangle in their keywords. Example, the width for a rectangle instance and square instance would both have the keyword _Rectangle__width. I get the prefix is due to me initializing all attributes as private attributes, but why doesn't the prefix change when it is a square instance. I assume for a square instance the width keyword should be _Square__width. Is it cause i am initializing the values for my square class using the super() function? Here is my code.
class Rectangle(Base):
    def __init__(self, width, height, x=0, y=0, id=None):
        super().__init__(id)
        self.__width = width
        self.__height = height
        self.__x = x
        self.__y = y

    def to_dictionary(self):
        return self.__dict__

class Square(Rectangle):
    def __init__(self, size, x=0, y=0, id=None):
        super().__init__(size, size, x, y, id)

    def to_dictionary(self):
        return self.__dict__

I tested the function in main using the following code.
r2 = Rectangle(5, 5, 5, 5)
rectangle_dict = r2.to_dictionary()
print(rectangle_dict)

s2 = Square(4, 4, 4)
square_dict = s2.to_dictionary()
print(square_dict)

The result was
{'id': 1, '_Rectangle__width': 5, '_Rectangle__height': 5, '_Rectangle__x': 5, '_Rectangle__y': 5}
{'id': 2, '_Rectangle__width': 4, '_Rectangle__height': 4, '_Rectangle__x': 4, '_Rectangle__y': 4}

All the keyword prefixes are the same. Why is that? And I know the to_dictionary method in my Square class is unnecessary as it could easily inherit the same from the Rectangle class, but the results would be the same as above.

Answer: Square.__init__ calls super() which calls Rectangle.__init__ and that's where the private variables are assigned. Private variables should only be used by the class that defines them. Inheritors have no business fiddling with them. That's literally why they exist. If you want inheritors like Square to see these variables, you shouldn't make them private to the class.