Myashka/SO-Python_QA-filtered-2023-tanh_score-after_2023_02 huggingface dataset:
Title: Is there a workaround for getting an executable to autorun in Windows 10?
Tags: python,windows,executable,autorun,inf

Question: I have an executable file I made using Python and I am trying to get it to run automatically upon being plugged in via a USB on an external drive. I've run into the issue that Windows seems to have removed that feature, but autorun is a key feature of what I am trying to create. Is there some workaround that has been found, or am I out of luck?
I tried making an INF file, here is what that looked like:
[autorun]
open=myfilename.exe
This doesn't work (meaning plugging in the USB acts as normal as opposed to starting up my executable) likely due to Windows disabling that feature past Windows 7.
One last thing is that it needs to work on any computer running Windows 10 with zero user input if at all possible. Is it time to go back to the drawing board?

Answer: This has been permanently revoked due to security problems. If you can find a way to do this, file a vulnerability report.
According to documentation, autorun still works on CD-ROMs but I don't trust it at all. If it works on USB CD-ROMS expect that to be taken away.
"One last thing is that it needs to work on any computer running Windows 10 with zero user input if at all possible." We don't want this anymore. It was a good idea in 1995. It was a bad idea in 1998. It was an unacceptable risk in 2005. It's gone in 2020.
Title: unit testing aws athena query
Tags: python,amazon-web-services,unit-testing,boto3

Question: import awswrangler as wr

def assume_role_and_query_athena(role_arn, query):
    # Assume the IAM role
    sts_client = boto3.client('sts')
    assumed_role = sts_client.assume_role(
        RoleArn=role_arn,
        RoleSessionName='AssumedRoleSession'
    )
    
    # Extract the temporary credentials from the AssumeRole response
    credentials = assumed_role['Credentials']
    access_key = credentials['AccessKeyId']
    secret_key = credentials['SecretAccessKey']
    session_token = credentials['SessionToken']
    
    # Use the temporary credentials to execute the query in Athena
    wr.athena.connect(
        access_key=access_key,
        secret_key=secret_key,
        session_token=session_token,
        region='us-east-1'  # Replace with your desired AWS region
    )
    df = wr.athena.read_sql_query(query)
    
    return df

My question is how do we unit test the above code. I don't feel like there is much to unit test for. The core test needed is probably about being able to establish the connection which cannot really be unit tested.

Answer: You could easily write a unit test for connection and to ensure that function when given sample inputs has the expected outputs (DF and whatever pipeline should now be set up). Especially if you have it as a part of your more extended tests instead of in CI as you're right it might be a little much for CI.
Title: What can i use better than ORB for matching photos
Tags: python,computer-vision,cbir

Question: I have folder named with a lot of location and each of them have 360 degree pics of each of those location i give it a random picture and it has to guess where am i, I tried ORB and it gave me good results and i made it better also but the issue is that lighting conditions might give me a hard time like sunny morning or cloudy morning as per some papers these parameters can cause a issue and i wanted to find is there some way that i can handle this issue is there any way of finding key points irrespective of the weather outside.
I tried to use ORB and getting good results for that movement of time but when lighting changes my dataset is not performing well so i need a solution for that

Answer: To handle the issue of lighting variations in your scenario, you can try the following approaches:

Histogram Equalization: Apply histogram equalization to normalize
the lighting conditions of your images. This technique redistributes
the pixel intensities to improve the contrast and make the images
more consistent across different lighting conditions.

Adaptive Thresholding: Instead of using a global threshold for image
binarization, consider using adaptive thresholding techniques. These
methods compute a local threshold for each pixel based on its
neighborhood, which can help handle variations in lighting.

Multi-Scale Analysis: Perform feature detection and matching at
multiple scales to capture different levels of details. By analyzing
the image at different scales, you can mitigate the impact of
lighting variations and improve the chances of finding matching
keypoints.

Image Enhancement Techniques: Apply image enhancement techniques,
such as gamma correction or contrast stretching, to improve the
visibility of details in the images and compensate for lighting
variations.

Explore Deep Learning-based Approaches: Deep learning models, such
as convolutional neural networks (CNNs), have shown promise in
handling variations in lighting conditions. You can consider
training a CNN on your dataset to learn robust features that are
less sensitive to lighting changes.
Title: Next quarter price when date-index is quarter end
Tags: python,pandas,datetime

Question: I am trying to calculate the abs change in price after a specific date. The data is quarterly and the end date is the last day of each quarter.
The code below works except where the date references the quarter ending in September. Then it returns null. It seems it would be easier to reference the next data point since the data is quarterly but when trying that, I get an indexing error.
quarterly_deviations_post_anomaly = {}

for key in anomaly_dfs.keys():
    quarterly_deviations_post_anomaly[key] = pd.DataFrame(columns=['1Q_abs_change'])
    
    anomaly_df = anomaly_dfs[key]
    q_df = q_data[key]
    
    for date in anomaly_df.index:
        if date in q_df.index and (date + pd.DateOffset(months=3)) in q_df.index:
            current_price = q_df.loc[date]['close']
            next_Q_price = q_df.loc[date + pd.DateOffset(months=3)]['close']
            deviation = abs((next_Q_price - current_price) / current_price)
            df = pd.DataFrame({'1Q_abs_change': deviation}, index=[date])
            quarterly_deviations_post_anomaly[key] = pd.concat([quarterly_deviations_post_anomaly[key], df])
        else:
            df = pd.DataFrame({'1Q_abs_change': float('NaN')}, index=[date])
            quarterly_deviations_post_anomaly[key] = pd.concat([quarterly_deviations_post_anomaly[key], df])

Answer: Your problem is there pd.DateOffset(months=3) on a date that is already the end of a quarter, if you start at the end of September and add 3 months, you'll get the end of December, this can cause errors.
Lets use pd.offsets.QuarterEnd(), it will offset to the end of the next quarter.
next_Q_price = q_df.loc[date + pd.offsets.QuarterEnd()]['close']
Title: What is the wildcard equivalent for boto3 dynamodb ProjectionExpression
Tags: python,amazon-dynamodb,boto3

Question: I have a function that calls boto3's query operation. My data object for each record is rather large so I have created an optional parameter that can be passed to reduce the fields that are returned. Here is the function:
def get_leads(type,leadTable,**kwargs):
    attributes = kwargs.get('attributes', None)
    client = boto3.resource('dynamodb')
    table = client.Table(leadTable)
    try:
        if attributes != None:
            response = table.query(ProjectionExpression=attributes,KeyConditionExpression=Key('lead_type').eq(type))
        else:
            response = table.query(KeyConditionExpression=Key('lead_type').eq(type))
        return response
    except Exception as e:
        print(e)
        return 0

This way, i can optionally pass attributes as a parameter
e.g.,
get_leads("user","table1",attributes="first_name,last_name,title,image_link")
returns only first_name,last_name,title,image_link fields
OR
get_leads("user","table1")
returns all fields
Ideally, I would like to simplify this function and set attributes to a wildcard ProjectionExpression value when it is not defined.
e.g.,
def get_leads(type,leadTable,**kwargs):
    attributes = kwargs.get('attributes', "*")
    client = boto3.resource('dynamodb')
    table = client.Table(leadTable)
    try:
        return table.query(ProjectionExpression=attributes,KeyConditionExpression=Key('lead_type').eq(type))
    except Exception as e:
        print(e)
        return 0

Sadly, I can't seem to find documentation anywhere stating what this wildcard value is or if it even exists.

Answer: There is no wild card, the absence of a ProjectipnExpression signifies DynamoDB to return all attributes.
Another tip is not to create a client in your function, create it globally and share its state across invocations, you'll get better latency and reduced costs if you use KMS at scale.
Title: How can i solve ImportError: Using the `Trainer` with `PyTorch` requires `accelerate>=0.20.1` when using Huggingface's TrainArguments?
Tags: python,nlp,importerror,huggingface-transformers,huggingface

Question: I'm using the transformers library in Google colab, and
When i am using TrainingArguments from transformers library i'm getting Import error with this  code:
from transformers import TrainingArguments

training_args = TrainingArguments(
    output_dir = "/content/our-model",
    learning_rate=2e-5,
    per_device_train_batch_size= 64,
    per_device_eval_batch_size = 16,
    num_train_epochs = 2,
    weight_decay = 0.01,
    evaluation_strategy = "epoch",
    save_strategy = "epoch",
    load_best_model_at_end = True,
    push_to_hub = False
)

This is the error i'm getting:
<ipython-input-28-0518ea5ff407> in <cell line: 2>()
      1 from transformers import TrainingArguments
----> 2 training_args = TrainingArguments(
      3     output_dir = "/content/our-model",
      4     learning_rate=2e-5,
      5     per_device_train_batch_size= 64,

4 frames
/usr/local/lib/python3.10/dist-packages/transformers/training_args.py in _setup_devices(self)
   1670         if not is_sagemaker_mp_enabled():
   1671             if not is_accelerate_available(min_version="0.20.1"):
-> 1672                 raise ImportError(
   1673                     "Using the `Trainer` with `PyTorch` requires `accelerate>=0.20.1`: Please run `pip install transformers[torch]` or `pip install accelerate -U`"
   1674                 )

ImportError: Using the `Trainer` with `PyTorch` requires `accelerate>=0.20.1`: Please run `pip install transformers[torch]` or `pip install accelerate -U 

I already tried pip install for 0.20.1 version of accelerate and pip install transformers[torch]
and both didn't worked.

Answer: I faced the same problem and followed the instructions of alvas.
I still had the issue, and I find that restarting runtime resolved the issue.
Title: R and julia not showing up on jupyter notebook
Tags: python,r,jupyter-notebook,julia

Question: I'm trying to install R and Julia on Jupiter notebook. I thought everything installed fine until I opened Jupiter notebook but there's only Python. I'm using Windows 11.
For R:
cd C:\Program Files\R\R-4.3.*\bin
R
install.packages("devtools")
# selected USA OR

devtools::install_github("IRkernel/IRkernel")
# didn't give me an option to select "1"

IRkernel::installspec()

q()

y

For julia:
julia
using Pkg
Pkg.add("IJulia")

I tried a couple of times but it just showed the location of the files.
I closed my admin command prompt and opened a regular one, tried Jupiter lab but there's only Python.

Answer: A Pkg.build("IJulia") can help here, it will make sure that the kernel is installed properly and afterwards you should be able to see it in Jupyter (maybe reload it once more).
Title: How to comment code blocks in Python that contain backslashes in strings?
Tags: python-3.x,comments

Question: A backslash in a Python string indicates the beginning of a control command, for e.g. text\n adds a new line after 'text'. If backslashes in strings shall not be interpreted as commands a r must precede the string. That is usually the case if the string contains a Windows directory, i.e. folder=r'C:\u'.
However, if I want to comment several lines by """ I get an error
"""
folder=r'C:\u'
"""

(unicode error) 'unicodeescape' codec can't decode bytes in position 13-14: truncated \uXXXX escape

If I comment line by line by # then there is no error, however this method is not comfortable for commenting a whole block:
#folder=r'C:\u'

How to comment a whole a block of code that contains strings with backslashes?
Python 3.10.9, IPython 8.7.0, Spyder 5.4.2, Windows 10
Edit
I am not looking for answers or comments in the sense "Don't use Windows, use Linux!"

Answer: Since your goal is to comment a block of code, the correct method is to precede each line with #.
When you do this, the parser ignores the rest of the line.  You won't need to add r and you don't need to worry about \.
This would be tedious as a manual process, but any decent editor will allow you to select a block and comment every line with a single keystroke.  (In Spyder, try Ctrl-1.  Other editors will typically use Ctrl-/ or a variation.)  Usually, the same key will uncomment a selected block.
The triple-quote method is not really a multi-line comment, but a multi-line string, which the interpreter is parsing.  Using it as a comment is not really 'Pythonic'.  You might run into other problems if, for example, the code block you are trying to comment includes triple-quoted strings.
Title: The tables cell size height and width not working in python docx
Tags: python,ms-word,docx,python-docx

Question: I have a document which has over 100 tables and I want to set the tables cell width to 0.54" and the cell height to 0.56". Although, the code has no problems but still the output doesn't have anything changed.
I tried searching for solutions but really didn't find anything I was not doing. I am sure that the code is working great. Here is the code
from docx import Document
from docx.shared import Inches


def resize_tables(document, width, height):
    for table in document.tables:
        for row in table.rows:
            for cell in row.cells:
                cell.width = width
                cell.height = height

# Load the Word document
doc = Document(r'C:\Users\Mosta\Downloads\S.docx')

# Specify the desired width and height for the cells
cell_width = Inches(0.54)
cell_height = Inches(0.56)

# Resize all tables
resize_tables(doc, cell_width, cell_height)

# Save the modified document
doc.save(r'C:\Users\Mosta\Downloads\document5.docx')

Answer: Have you tried setting the autofit property in each table to False?
You might have to set the height for the row, not per each cell, and apply the proper WD_ROW_HEIGHT_RULE per row. Also try setting the width per column intead of per cell.
Alternatively, try setting width value in EMU rather than Inches.
Title: FileNotFoundError while importing class that reads csv
Tags: python,csv,directory

Question: I'm working on a project that has this structure:

src/
data/ with csv and txt files
mod/ with python classes
main.py

classes in mod/ use read csv to get data in form of
class Currency:
    def __init__(self, country_name: str):
        self.country = country_name
        self.code = self.get_code()

    def get_code(self) -> str:
        with open("../data/iso-4217.csv") as file:
            country_list = list(csv.reader(file))
            for row in country_list[1:]:
                if row[0] == self.country:
                    code = row[1]
        return code

And it works just fine. But whenever I import the class itself to main.py, I get
FileNotFoundError: [Errno 2] No such file or directory: '../data/iso-4217.csv'
Is there a way to make correct import using relative pathing?
Using absolute pathing seems to fix the issue, but i'd rather use relative

Answer: Try replacing "../data/iso-4217.csv" by  "data/iso-4217.csv" as you are working in the main.py directory's.
Yet another option is to send the relative path in a variable to make the class usable from anywhere.
Title: Managing monorepo versioning using modified version of semver in a python project
Tags: python-3.x,microservices,monorepo,semantic-versioning

Question: Consider you have a monorepo in github, whereas each folder represents a microservice.
Now I want to be able to have a git tag for each microservice separately.
Since we are sharing one repo, each tag for example v1.0.0 for microservice foo1 will capture this tag from being created for microservice foo2.
Instead, we can create tags named foo1-v1.0.0 and foo2-v1.0.0 to solve this collision.
My question if we have some nice package or tool for doing precisely this convention- meaning semver with some prefix?
Thanks in advance

Answer: If your package feed is fully SemVer compliant, you can use the build metatag:

Package the microservices separately.
Apply the commit hash from which they were built to the metatag in the package version string (v1.0.42+commitId)
Tag the commit hash with the package name and full version string (MicroserviceA v1.0.42+commitId).

Sadly, not all package services (NPM comes to mind) are SemVer compliant and often drop the metatag, arguing that it is not used for sorting. Also, not all version automation can do repo tagging with a prefix other than 'v'.
Title: ansible-lint failing with: AttributeError: 'str' object has no attribute 'resolve'
Tags: python,python-3.x,ansible,ansible-lint

Question: While running a github linter workflow on a new CentOS VM that I set up, I got the following error:
Run ansible-lint --parseable ./ansible
Traceback (most recent call last):
  File "/usr/local/bin/ansible-lint", line 8, in <module>
    sys.exit(_run_cli_entrypoint())
  File "/usr/local/lib/python3.9/site-packages/ansiblelint/__main__.py", line 344, in _run_cli_entrypoint
    sys.exit(main(sys.argv))
  File "/usr/local/lib/python3.9/site-packages/ansiblelint/__main__.py", line 197, in main
    initialize_options(argv[1:])
  File "/usr/local/lib/python3.9/site-packages/ansiblelint/__main__.py", line 110, in initialize_options
    options.cache_dir = get_cache_dir(options.project_dir)
  File "/usr/local/lib/python3.9/site-packages/ansible_compat/prerun.py", line 13, in get_cache_dir
    basename = project_dir.resolve().name.encode(encoding="utf-8")
AttributeError: 'str' object has no attribute 'resolve'
Error: Process completed with exit code 1.

I tried with the following package versions which all failed with the above error:

python3: 3.9.16
ansible: 8.0.0 or 6.6.0
ansible-compat:      4.1.2 or 2.2.7
ansible-core:        2.15.0 or 2.13.9
ansible-lint:        6.17.0 or 6.10.2

Using pip, the following dependencies were also installed:

flake8
yamllint==1.28.0
codespell
comment_parser

Is there some linter dependency or something else that I am missing?
The step within the linter job is:
 - name: 'Lint yaml - ansible, GH actions and GH workflows'
   run: yamllint -f parsable ./
   if: always()

Answer: In my environment, downgrading ansible_compat to v3.0.1 with pip install solved the problem. From v4 on, prerun.py contains the line that gives error.
Title: DJANGO non nullable field migration issue
Tags: python,django,unit-testing

Question: Im a DJANGO/python novice. I have added two fields so that changes in enterprise and DJANGO admin are logged as who updated them and what date it was updated.
the constraint is that these fields cannot be null.
my changes are as follows:
   class Enterprise(models.Model):
    last_updated = models.DateTimeField(
        null=False,
        auto_now=True,
        verbose_name='Last Updated Date',
        help_text="Points to time this was last modified",
    )

    last_updated_by = models.CharField(
        null=False,
        max_length=100,
        default='N/A',
    )

AND
class EnterpriseAdmin(admin.ModelAdmin):
    model = Enterprise

    readonly_fields = (
        "last_updated",
        "last_updated_by",)

    fieldsets = ('last_updated','last_updated_by',),


however when I run pytest, I'm receiving an error: null value in column "last_updated" violates not-null constraint.
when I ran makemigrations, Django prompted me to add a default for last_updated:
You are trying to add a non-nullable field 'last_updated' to enterprise without a default; we can't do that (the database needs something to populate existing rows). Please select a fix: 1) Provide a one-off default now (will be set on all existing rows) 
so I chose option '1' and default=datetime.datetime(1970, 1, 1, 0, 0),preserve_default=False, is added to last_updated in my migration file.
    migrations.AddField(
            model_name='enterprise',
            name='last_updated',
            field=models.DateTimeField(auto_now=True, default=datetime.datetime(1970, 1, 1, 0, 0), help_text='Points to time this was last modified', verbose_name='Last Updated Date'),
            preserve_default=False,
        ),

I was under the impression that the Unix epoch would be added as default in all existing columns.

Answer: When adding a new field to a model with an existing table Django needs a value to use for all the pre-existing rows.
If the field has a default specified that will be used and everyone is happy because that's nice and straightforward.
If the field is nullable and doesn't have a default value Django will set the column to null for existing rows. This obviously doesn't work for a non-nullable field.
Lacking either of those solutions Django asks you what it should do. It doesn't quietly set a non-null value on any field since that might have unintended effects.
Maybe you already have a command that checks enterprises, foos, and bars which runs every night. It uses a getattr to determine if the object has a last_updated attribute and deletes all the entries which were last updated twenty years ago or more. Setting the default value to 1970 in that case could unintentionally delete a large part of your database.
Title: Impossible Python Script Not Generating Excel File
Tags: python,excel,web-scraping

Question: I've be struggling with this Python script for a few days. I'm trying to scrape data and generate an excel file as an output. I don't get any error messages when I run the code, but the excel file isn't created.
I'm on a Windows 10 os, running script in Sublime Text with the Python Build, I've installed all of the packages in the script, made sure I have permission to write to the CD, but nothing I've tried seems to work. Any ideas? Thanks!
import pandas as pd
from bs4 import BeautifulSoup
import requests

url = "https://www.fangraphs.com/leaders.aspx?pos=all&stats=bat&lg=all&qual=y&type=1&season=2023&month=0&season1=2023&ind=0&team=0&rost=0&age=0&filter=&players=0&startdate=2023-01-01&enddate=2023-12-31&page=1_500"

html_content = requests.get(url).text

soup = BeautifulSoup(html_content, "lxml")

data = {}

leaders = soup.find("table", attrs={"class": "rgMasterTable"})
leaders_data = leaders.tbody.find_all("tr")

headings = []

for td in leaders_data[0].find_all("td"):
    if td.b:
        headings.append(td.b.text.replace('\n', '').strip())

for table, heading in zip(leaders_data[1].find_all("table"), headings):
    t_headers = []
    for th in table.find_all("th"):
        t_headers.append(th.text.replace('\n', '').strip())

    table_data = []
    for tr in table.tbody.find_all("tr"):
        t_row = {}
        for td, th in zip(tr.find_all("td"), t_headers):
            t_row[th] = td.text.replace('\n', '').strip()
        table_data.append(t_row)
    data[heading] = table_data

for topic, table in data.items():
    df = pd.DataFrame(table)

    excel_filename = "C:/Users/(myuser)/Documents/rake_list.xlsx"
    writer = pd.ExcelWriter(excel_filename, engine='xlsxwriter')
    df.to_excel(writer, index=False)
    writer.save()

import os
print(os.path.isfile("rake_list.xlsx")) 
#I've added this to see if it has saved elsewhere but it return "False"

I've tried changing the directory, generating a csv file instead of an excel file, running as a command prompt outside of Sublime Text, but the excel file will not generate.

Answer: It looks like both data and headings are empty, so it simply isn't writing a blank file. I think you need to revisit how you're parsing the website to ensure you're properly building your data object
Title: How do calculators handle floating point errors?
Tags: python,floating-point,precision

Question: I am writing a calculator interface in Python using Tkinter. I am aware of the floating point precision issues that can arise in computers, but I have never had to deal with them directly.
In my calculator for example, I am multiplying two floats together, say 3.2 and 3.0. Instead of resulting in 9.6, however, I get 9.60000000000001 (or something). Normally, I would just round this number. However, under circumstances where I would need that extra precision, I wish to keep it. What is the standard practice for dealing with this issue?

Answer: However, under circumstances where I would need that extra precision, I wish to keep it.

Your precision needs exceed what is readily available with standard binary floating point.
Use decimal or use higher precision binary math to reduce the occurrence of the perceived imprecision.

What is the standard practice for dealing with this issue?

Common calculators always use more internal precision (decimal or not) than they show to avoid such floating point artifacts.
I recommend internal math that is at least 2-3 decimal places (8-10 bits) more than the significant maximum number of digits displayed.
Title: Guarantee asyncio execution order
Tags: python,python-asyncio

Question: I have seen other answers on here stating that asyncio doesn't guarantee execution order, only that the order of the outputs will match that of the inputs.
Is there a way I can guarantee the execution order
For example, if I have a list of the functions I want to run, will calling create task on each of them before calling gather work? Is there something else I can do to achieve this?

Answer: We can use asyncio.ensure_future() instead of asyncio.create_task() to create the tasks. This makes sure that tasks are scheduled in the order they were created. Once they are scheduled in order we can use await asyncio.gather(*task_list)
to execute tasks in the order they are created.

However ensure_future() does not guarantee the exact execution order if tasks have different durations. It ensures the order they are scheduled but execution still depends on various other factors.

The best way should be to use await  keyword to await the completion of each task before moving on to the next one. This will guarantee execution order but they will be executed sequentially.
If you require more fine-grained control, you can(use locks) lock a critical section so that only one task can execute the critical section at a time.
Title: Efficient ways of handling dataframes with columns that are mutually exclusive?
Tags: python,pandas,csv

Question: As an example imagine I have a csv below
id | a1 | a2 |....| aN | b1 | b2 |....| bM |
___________________________________________
0  |data|data|....|data|None|None|....|None|
1  |data|data|....|data|None|None|....|None|
2  |data|data|....|data|None|None|....|None|
3  |None|None|....|None|data|data|....|data|
4  |None|None|....|None|data|data|....|data|
....

I have N a columns and M b columns, and the a columns and b columns are mutually exclusive i.e. if I have data in a then I wouldn't have anything in b. data in this case is mostly string or float values.
It feels inefficient that I will have M or N elements for each row that don't contain anything.
I can split the above into two different dataframes i.e.
df_a
id | a1 | a2 |....| aN |
________________________
0  |data|data|....|data|
1  |data|data|....|data|
2  |data|data|....|data|


df_b
id | b1 | b2 |....| bM |
________________________
3  |data|data|....|data|
4  |data|data|....|data|

But then I have two dataframes that I have to keep track of instead of one dataframe.
What is the most efficient way of keeping the data together without creating a bloated dataframe? Would the solution work if I have c and d columns as well?
One thing I can do is make the csv into an excel sheet and put the b columns in a different sheet. But it's still a bit clunky for me.

Answer: @kkawabat, how about this, first separate a,b then assign both dataframes, a new row called, 'TYPE', for df of a fill TYPE with 'a' and TYPE 'b' for other and then merge both data frame, in such case u don't need to make df columns like a1, a2, b1, b2, you can just do col1,col2,.... or also u can set that TYPE to boolean and refer True as a and False as b.
Title: How to create .cdx file from .dbf file in pyhton?
Tags: python,visual-foxpro,dbf

Question: import dbf
open file
table = dbf.Table(str("/art_gr.dbf"))
open table
table.open(mode=dbf.READ_WRITE)
add data to table
table.append({'DENUMIRE': '2', 'COD': '3'})
open cdx file
index = dbf.Idx(filename='C:/SAGA C.3.0/salv_bd/0001/art_gr.cdx', table=table)
#how can i update the index with new data??
close table
table.close()
I want to update an existing .cdx file with data inserted in .dbf file

Answer: My python dbf library doesn't support index files at this time.  If no other means of updating the .cdx index file is available, you'll need to rebuild the indices in the accounting program.
Title: Is a UDP broadcast actually a unicast?
Tags: python,udp,broadcast

Question: If one has to specify a certain port for sending a broadcast, how is it different to a unicast?
Isn't the idea of a broadcast, that you can send a packet without specifying its destination in advance?
Example code for a broadcast in python:
def send_udp_broadcast(message, port):
 # Create a UDP socket
 udp_socket = socket.socket(socket.AF_INET, socket.SOCK_DGRAM)

 # Enable broadcasting mode
 udp_socket.setsockopt(socket.SOL_SOCKET, socket.SO_BROADCAST, 1)

 # Set the broadcast address
 broadcast_address = '<broadcast>'
 server_address = (broadcast_address, port)
 udp_socket.sendto(message.encode(), server_address)

I tried performing a broadcast without specifying a port, but it seems not possible.

Answer: TL;DR Addresses determine which machine should see the packet. Ports determine which service on a machine should see the packet.

When a packet is delivered to a network containing the destination, one of two things can happen:

A router can send the packet directly to the interface indicated by the destination address.
The packet is available to anyone on that network, but an interface is supposed to disregard any packet whose destination address does not match its own address.

In either case, using a broadcast address causes

The router to deliver the packet to every interface on the network
Each interface to recognize the packet as intended for it.

Once an interface has a packet, the port is used to determine who on the machine should see the packet.
Title: Pandas read_json script that used to work now produces an error
Tags: python,pandas

Question: I have a script that up until recently worked fine, but is now producing an error.
import requests
import pandas as pd

# Set the url to given endpoint
url = "https://SomeURL/SomeEndpoint"
print('URL set')

# Connect to endpoint with credentials and put results in dictionary
URLresponse = requests.get(url,auth=("SomeUser", "SomePassword"), verify=True)
print('connection to endpoint')

# Load the response as proper JSON into a var
rawdata = (URLresponse.content)
print(type(rawdata))
print('populating variable')

# print(rawdata)

# Load the var into a dataframe
df = pd.read_json(rawdata) 
print('load variable into df')

print(df)

This used to work fine but now it is producing an error as following:
  File "C:\Program Files\Python310\lib\site-packages\pandas\io\common.py", line 901, in get_handle
    raise TypeError(
TypeError: Expected file path name or file-like object, got <class 'bytes'> type

How can I go ahead to troubleshoot this?

Answer: You can change df = pd.read_json(rawdata) to df = pd.read_json(io.StringIO(rawdata.decode('utf-8')))
You will need to include import io earlier in your file as well.
Title: Question about python functions and access to variables
Tags: python,function

Question: I am currently learning Python and looking at how to define and use functions. But I have a question about something that I find confusing.
In the tutorial there is something like this:
alphabet = ['a','b','c']
text = input("type your message")
shift = int(input("type the number of characters to shift message by"))

def encrypt(txt_msg, shft_msg):
    for letter in txt_msg:
        position = alphabet.index(letter)
        # rest of code...
    print(result)

encrypt(text, shift)

So I understand when I call encrypt(text,shift) I am passing those arguments to the parameters in the function txt_msg and shft_msg, so that they are available within the function.
So why then have they not also passed alphabet as well? In my mind, based on experience with MATLAB, if you want variables from the main workspace to be available for the function, you have to pass them as arguments. Why is alphabet available to the function without passing it as an argument? If that is the case, what is the point of passing any arguments?

Answer: Alphabet is a global variable. You may use any global variable inside your function.
So Whats the point of passing arguments then?
The point of passing arguments is that you can use the values that have not been defined yet at the time  you're creating the function.
The value of txt_msg and  shft_msg
is not known when you are defining the function. Unless you write these as arugments there is no way for you to write the function and then provide the values afterwards
Title: How to create cache in opcua asycio file
Tags: python,python-asyncio,opc-ua

Question: I want to create cache file for opcua server using opcua-asyncio, I have looked at opcua-asyncio github repository , but I did not understand enough, it was mentioned that cache can be created but nobody mentioned how can be cache created. so I am wondering how can I create cache file for optimize the performance of my python file. here is the code I am using  :
import asyncio

import logging import sys from asyncua.common.ua_utils import value_to_datavalue

sys.path.insert(0, "..")

from asyncua import Server

async def main(): logging.basicConfig(level=logging.INFO) server = Server()

await server.init()


server.set_endpoint("opc.tcp://localho:4140")

await server.import_xml("Opc.Ua.Di.Nodeset2.xml")
await server.import_xml("Opc.Ua.Machinery.Nodeset2.xml")
await server.import_xml("Opc.Ua.Pumps.Nodeset2.xml")
await server.import_xml("custom.Nodeset.xml")


#starting!
async with server:
    while True:
        await asyncio.sleep(1)

if name == "main":

asyncio.run(main())

Could someone please provide a solution?
What is the easiest way to create a cache file?
FYI I want to run this file into raspberry pi!
I tried several ways but unfortunately, neither of them worked.
your advice and solutions would be appreciated.

Answer: I think the feature was removed because of safety concerns or just got lost in the port to asyncio.
So currently this feature doesn't exist.
Title: pytorch dataloader- RuntimeError: stack expects each tensor to be equal size
Tags: python,pytorch,artificial-intelligence,resnet,torchvision

Question: I'm trying resnet implementation from scratch.
This error occurred during training after setting the dataset in Dataloader:
RuntimeError: stack expects each tensor to be equal size, but got [3, 224, 224] at entry 0 and [1, 224, 224] at entry 25

Usually, I know why this message displayed. It's about unequal image dataset
But before I got this message, I apply transforms so I didn't expect error like this.
Sometimes the message [4,224,224] appears instead of [1,224,224].
Of course, I checked dataset all images are color
Here's the code
## Class declaration, transforms codes are applied __getitem__()

class cnd_data(torch.utils.data.Dataset):
    def __init__(self, file_path, train=True, transforms=None):

        self.train=train
        self.transforms=transforms

        self.cat_img_path=os.path.join(file_path, 'data\kagglecatsanddogs\PetImages\Cat')
        self.dog_img_path=os.path.join(file_path, 'data\kagglecatsanddogs\PetImages\Dog')
        
        self.cat_list=natsort.natsorted(glob.glob(self.cat_img_path + '/*.jpg'))
        self.dog_list=natsort.natsorted(glob.glob(self.dog_img_path + '/*.jpg'))
 

        if self.train==True:
            self.imgn_list=self.cat_list[:12000]+self.dog_list[:12000]
            self.img_label=[0]*12000+[1]*12000

        else:
            self.imgn_list=self.cat_list[12000:]+self.dog_list[12000:]
            self.img_label=[0]*500+[1]*500

    def __len__(self):
        return len(self.img_label)

    def __getitem__(self, idx):

        image_data=Image.open(self.imgn_list[idx])

        print(self.imgn_list[idx])
        if self.transforms:
            sample=self.transforms(image_data)

        return sample, self.img_label[idx]


## And transforms setting codes
transforms=transforms.Compose([
    transforms.Resize(size=(224, 224)),
    transforms.ToTensor()])

Answer: Um... I checked that the problem was that the code I wrote when I read the image from PIL was not able to read the color image with bit-depth 8 as desired. If bit-depth is 8, this is because PIL.Image.open() recognizes and treats the channel as one.
When the data set was created, the data were collected in various forms, so it seems that although most of the data were 24-bit color images, it contained some 8-bit data (if the channel was 4, it was a 32-bit color image, because RGB added alpha channel was added)
Anyway, this problem will be solved by checking the function that converts it to three channels if it is entered in 8-bit color. I'll have to look at the data more carefully from now on. Thank you to everyone who commented.
Title: Why is this the default datetime in Python?
Tags: python,datetime

Question: I tried on an online Python interpreter
from datetime import datetime
print(datetime.fromtimestamp(0))

And it predictably prints 1970-01-01 00:00:00. I know this is the Unix Epoch. However, I tried it on my machine and it prints 1969-12-31 19:00:00. What could be the reason for this?

Answer: this is really asking why does that function apply the time zone. The implementation of that method uses the libc's localtime function, the man page of which say this:

The localtime() function converts the calendar time timep to broken-down time representation, expressed relative  to  the user's specified timezone.  The function acts as if it called tzset(3) and sets the external variables tzname with information about the current timezone

The system call is automatically applying the local time zone, if it's known. Some hosts have their timezone set to UTC.
Title: Is there a way to access feature names/labels from a keras model alone?
Tags: python,tensorflow,machine-learning,keras,feature-engineering

Question: I'm trying to retrieve feature names from a Keras model in a generalized way. I want to load a pretrained model and obtain its feature names, like this:
labels = model.get_feature_names()

I'm looking for something that works with any Keras model, ideally as a method that takes a black box Keras model and returns the feature names that the model operates on.
For example, in SKlearn I can do this:
from sklearn.base import BaseEstimator
if isinstance(model, BaseEstimator):
  return lambda m: {
    'labels': m.feature_names_in_,
  }

Is there a Keras/TensorFlow equivalent? Any insight is greatly appreciated.

Answer: No, unlike sklearn there isn't a Keras/Tensorflow equivalent for obtaining feature names. This is because tensorflow-keras models focus more on the shape and dimensions of the input tensors rather than the individual features. The model utilizes the input features as tensors, regardless of name.
With some effort you might be able to track your individual features by values. However, if your end objective is building a classifier out of tabular data (and not some complex Computer-Vision or NLP tasks), you can consider using alternatives such as XGBoost (or perhaps other classifiers from sklearn itself). These classifiers are feature focused and you'll be able to work with the feature_names. They are also better suited to tabular data due to their versatility and ability to generalize well even with limited samples/features.
Title: Randomly generate all unique pair-wise combination of elements between two list in set time
Tags: python,random

Question: I have two lists:
a = [1, 2, 3, 5]
b = ["a", "b", "c", "d"]

And would like to generate all possible combinations with a python generator. I know I could be doing:
combinations = list(itertools.product(a,b))
random.shuffle(combinations)

But that one has an extreme memory cost as i would have to hold in memory all possible combinations, even if only wanted two random unique combinations.
My target is to get a python generator that has its memory cost increase with the more iterations are requested from it, getting to the same O memory cost as itertools at max iterations.
I had this for now:
def _unique_combinations(a: List, b: List):
    """
    Creates a generator that yields unique combinations of elements from a and b
    in the form of (a_element, b_element) tuples in a random order.
    """
    len_a, len_b = len(a), len(b)
    generated = set()
    for i in range(len_a):
        for j in range(len_b):
            while True:
                # choose random elements from a and b
                element_a = random.choice(a)
                element_b = random.choice(b)
                if (element_a, element_b) not in generated:
                    generated.add((element_a, element_b))
                    yield (element_a, element_b)
                    break

But its flawed as it can theoretically run forever if the random.choice lines are unlucky.
I'm looking to modify that existing generator so it generates the indexes randomly within a fix set of time, it will be okay to keep them track of as this will be linear increase in memory cost and not exponential.
How could i modify that random index generator to be bound in time?

Answer: @gog:
The code snippet has a limitation in terms of scalability. It is utilizing a set to track generated combinations, the memory usage and performance become problematic as the total number of possible combinations increases.
Title: Tkinter throws ImportError
Tags: python,tkinter

Question: I was trying to make my first Tkinter project in Python but it just shows me this:
>>> from tkinter import *
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/usr/lib/python3.11/tkinter/__init__.py", line 38, in <module>
    import _tkinter # If this fails your Python may not be configured for Tk
    ^^^^^^^^^^^^^^^
ImportError: libtk8.6.so: cannot open shared object file: No such file or directory

I am using Arch.
I have tried pip install tkinter but it didn't help.
Pls help

Answer: Try # pacman -S tk I had the same problem and that fixed it.
Title: cppyy crash with uint8_t arrays
Tags: python,c++,c,cppyy

Question: With cppyy 2.3.1 and python3.8, I could do the following:
import cppyy
cppyy.cppdef(''' struct T { uint8_t t[10]; }; ''')
t = cppyy.gbl.T()
bytes(t.t)

With cppyy 2.4.0 and above, I get a sigsev:

#8  signal handler called
#9  __memmove_avx_unaligned_erms () at ../sysdeps/x86_64/multiarch/memmove-vec-unaligned-erms.S:384
#10 0x000056085fd8d582 in memcpy (__len=2147483640, __src=, __dest=0x7f0f27d56030) at
/usr/include/x86_64-linux-gnu/bits/string_fortified.h:34
#11 PyBuffer_ToContiguous (buf=buf entry=0x7f0f27d56030, src=src entry=0x7ffc116933b0, len=2147483640, order=order entry=67 'C') at
Objects/memoryobject.c:997
#12 0x000056085fe188d9 in _PyBytes_FromBuffer (x=) at Objects/bytesobject.c:2689
#13 0x000056085fd56279 in bytes_new_impl (errors=0x0, encoding=0x0, x=0x7f0fa7dba6b0, type=0x56086000bac0 <PyBytes_Type>) at
Objects/bytesobject.c:2667
#14 bytes_new (type=0x56086000bac0 <PyBytes_Type>, args=, kwargs=) at Objects/clinic/bytesobject.c.h:894
#15 0x000056085fda0bd5 in type_call (type=type entry=0x56086000bac0 <PyBytes_Type>, args=args entry=0x7f0fa7df0c70, kwds=kwds entry=0x0)
at Objects/typeobject.c:1100

When I simply want to print t.t[0] I get a random number above 255. Trying to set it has no effect.
I've tried by installing from pip and from sources, and with both python 3.8 and 3.11.
cppyy-2.4.0 changelog mentions something about uint8_t enums, that were previously treated as 1-character strings.
Interestingly enough, if I use "unsigned char" instead of "uint8_t", it works as expected; however I have plenty of uint8_t in my code, so it's not an option.
Can anyone confirms this behavior ?
If it's indeed a bug in cppyy, is there another way or a workaround to access the buffer?

Answer: This particular case is now fixed in repo.
The problem when mapping int8_t or uint8_t is that these are supposed to behave as integer types, but the actual underlying C++ types are signed char and unsigned char. Whereas in C/C++ the difference between char and int types is minimal, Python's str and int are completely unrelated types. Iow., wherever they are used, the int8_t and uint8_t typedefs must not be resolved, while at the same time, it's not possible to distinguish between e.g. int8_t and signed char in C++ itself using overloads or RTTI. Bit of a conundrum.
Title: Swagger Documentation for Python Pyramid Application
Tags: python,swagger,openapi,pyramid

Question: I have an old python pyramid web application and I just wanted to create an API documentation (specifically Swagger or OpenAPI3) for it. Is there a way to do that without writing the entire swagger YAML schema yourself. Is there a library that automatically scans through all the @view_config routes in the application and then generates the document for it.
There are similar things in spring or spring-boot where the swagger schema/spec is autogenerated based on the code. Is there a similar of doing this for the python pyramid application.
PS. The application uses Poetry as the dependancy manager.

Answer: As far as I know, there isn't OpenAPI-sec generating solution for Pyramid. pyramid_openapi reads Yaml file and generates view bindings for you, not the opposite what you are asking in the question.
My guess is that the best solution would be lift some code from FastAPI and integrate that to your Pyramid application.
Title: Does python redefine guess of variable scope inside function?
Tags: python,function,scope,global,local-variables

Question: I have a simple example that is confusing me, concerning local and global scopes in functions.
a=1
def func():
    print(a)   # Error: local variable 'a' referenced before assignment
    a=2
    print(a)

func()

If I comment the first print statement it runs fine. The local "a" takes precedence over the global "a" and hence 2nd print gives 2.
If I comment the a=2, then the print statements in the local scope see the global scope and print a=1.
What is confusing me is that when I run the entire function python throws an error: "local variable 'a' referenced before assignment"
What I was expecting was that the first print(a) produces 1 and the second print(a) produces 2.
I'm struggling to tease this apart. I examined locals() inside the func() so the first print function sees no local but it does see a global a. I'm really a bit lost and would appreciate someone explaining why it appears that the combination errors when the individual bits work fine.

Answer: When you make an assignment to a variable in a scope, it becomes local to that scope and shadows any similarly named variable in the outer scope. Since Python's scoping is static, wheter a is local or global is determined at compile-time and that's why, when it comes to func(), Python sees a is assigned within the function body and  treats it as a local variable throughout the entire function. When you try to print(a) before local a has been assigned a value within the function, you will face an error. When you comment out a = 2, then a is considered global and print() executes successfully.