eshangj/stackoverflow_q_and_a_sample huggingface dataset:
Question: With GitPython, I can create a new repo with the following: from git.repo.base import Repo Repo.init('/tmp/some-repo/')  The repo is created with the default branch master. How can I modify this default branch? Update: As suggested in the answers below, I have tried using Repo.init('/tmp/some-repo', initial_branch="main"), however it renders this exception: Traceback (most recent call last): File "/app/checker/tests.py", line 280, in test_alternative_compare_branch comp_repo_main = Repo.init( File "/usr/local/lib/python3.9/site-packages/git/repo/base.py", line 937, in init git.init(**kwargs) File "/usr/local/lib/python3.9/site-packages/git/cmd.py", line 542, in <lambda> return lambda *args, **kwargs: self._call_process(name, *args, **kwargs) File "/usr/local/lib/python3.9/site-packages/git/cmd.py", line 1005, in _call_process return self.execute(call, **exec_kwargs) File "/usr/local/lib/python3.9/site-packages/git/cmd.py", line 822, in execute raise GitCommandError(command, status, stderr_value, stdout_value) git.exc.GitCommandError: Cmd('git') failed due to: exit code(129) cmdline: git init --initial-branch=main stderr: 'error: unknown option `initial-branch=main'  In the git docs, it states that the command for setting initial branch is --initial-branch (https://git-scm.com/docs/git-init/2.28.0#Documentation/git-init.txt---initial-branchltbranch-namegt). Judging by the error, I think that the additional kwargs feature of GitPython is not including the -- prefix.

Answer: According to the docs, init takes the same arguments as git init as keyword arguments. You do have to turn - into _. from git import Repo Repo.init('/tmp/some-repo/', initial_branch='main')   UPDATE initial-branch was added very recently in v2.28.0. You'll need to upgrade Git to use it. If you can't, manually change the branch name with branch.rename(new_name). Unfortunately you can't do this until after the first commit, no branches truly exist yet. That's a Git limitation and why they added initial-branch and also the init.defaultBranch config option.
Question: I am using flake8 (with flakehell but that should not interfere) and keep its configuration in a pyproject.toml file. I want to add a per-file-ignores config but nothing works and there is no documentation on how it is supposed to be formatted in a toml file. Flake8 docs show only the 'native' config file format: per-file-ignores = project/__init__.py:F401 setup.py:E121 other_project/*:W9  There is no description / example for pyproject.toml. I tried: per-file-ignores=["file1.py:W0621", "file2.py:W0621"]  and per-file-ignores={"file1.py" = "W0621", "file2.py" = "W0621"}  both of which silently fail and have no effect (the warning is still raised). What is the proper syntax for per-file-ignores setting in flake8/flakehell while using pyproject.toml?

Answer: flake8 does not have support for pyproject.toml, only .flake8, setup.cfg, and tox.ini  disclaimer: I am the flake8 maintainer
Question: Given the DataFrame generated by: import numpy as np import pandas as pd from datetime import timedelta np.random.seed(0) rng = pd.date_range('2015-02-24', periods=14, freq='9H') ids = [1]*5 + [2]*2 + [3]*7 df = pd.DataFrame({'id': ids, 'time_entered': rng, 'val': np.random.randn(len(rng))})  df:  id time_entered val 0 1 2015-02-24 00:00:00 1.764052 1 1 2015-02-24 09:00:00 0.400157 2 1 2015-02-24 18:00:00 0.978738 3 1 2015-02-25 03:00:00 2.240893 4 1 2015-02-25 12:00:00 1.867558 5 2 2015-02-25 21:00:00 -0.977278 6 2 2015-02-26 06:00:00 0.950088 7 3 2015-02-26 15:00:00 -0.151357 8 3 2015-02-27 00:00:00 -0.103219 9 3 2015-02-27 09:00:00 0.410599 10 3 2015-02-27 18:00:00 0.144044 11 3 2015-02-28 03:00:00 1.454274 12 3 2015-02-28 12:00:00 0.761038 13 3 2015-02-28 21:00:00 0.121675  I need to, for each id, remove rows which are more than 24hours (1 day) from the latest time_entered, for that id. My current solution: def custom_transform(x): datetime_from = x["time_entered"].max() - timedelta(days=1) x = x[x["time_entered"] > datetime_from] return x df.groupby("id").apply(lambda x: custom_transform(x)).reset_index(drop=True)  which gives the correct, expected, output:  id time_entered val 0 1 2015-02-24 18:00:00 0.978738 1 1 2015-02-25 03:00:00 2.240893 2 1 2015-02-25 12:00:00 1.867558 3 2 2015-02-25 21:00:00 -0.977278 4 2 2015-02-26 06:00:00 0.950088 5 3 2015-02-28 03:00:00 1.454274 6 3 2015-02-28 12:00:00 0.761038 7 3 2015-02-28 21:00:00 0.121675  However, my real data is tens of millions of rows, and hundreds of thousands of unique ids, because of this, this solution is infeasible (takes very long time). Is there a more efficient way to filter the data? I appreciate all ideas!

Answer: Generally, avoid groupby().apply() since it's not vectorized across groups, not to mention the overhead for memory allocation if you are returning new dataframes as in your case. How about finding the time threshold with groupby().transform then use boolean indexing on the whole data: time_max_by_id = df.groupby('id')['time_entered'].transform('max') - pd.Timedelta('1D') df[df['time_entered'] > time_max_by_id]  Output:  id time_entered val 2 1 2015-02-24 18:00:00 0.978738 3 1 2015-02-25 03:00:00 2.240893 4 1 2015-02-25 12:00:00 1.867558 5 2 2015-02-25 21:00:00 -0.977278 6 2 2015-02-26 06:00:00 0.950088 11 3 2015-02-28 03:00:00 1.454274 12 3 2015-02-28 12:00:00 0.761038 13 3 2015-02-28 21:00:00 0.121675
Question: Is there a way to disallow overriding given keyword arguments in a partial? Say I want to create function bar which always has a set to 1. In the following code: from functools import partial def foo(a, b): print(a) print(b) bar = partial(foo, a=1) bar(b=3) # This is fine and prints 1, 3 bar(a=3, b=3) # This prints 3, 3  You can happily call bar and set a to 3. Is it possible to create bar out of foo and make sure that calling bar(a=3, b=3) either raises an error or silently ignores a=3 and keeps using a=1 as in the partial?

Answer: This is by design. The documentation for partial says (emphasize mine):  functools.partial(func, /, *args, **keywords)  Return a new partial object which when called will behave like func called with the positional arguments args and keyword arguments keywords. If more arguments are supplied to the call, they are appended to args. If additional keyword arguments are supplied, they extend and override keywords.  If you do not want that, you can manually reject frozen keyword arguments: def freeze(f, **kwargs): frozen = kwargs def wrapper(*args, **kwargs): kwargs.update(frozen) return f(*args, **kwargs) return wrapper  You can now do: >>> baz = freeze(foo, a=1) >>> baz(b=3, a=2) 1 3
Question: I am building a Python web scraping script and i have to use cv2 (OpenCV). So I install using pip install opencv-python as the website directs. And it also installs numpy as a dependency. However, right after installing that, I'm unable to run my python script. It crashes with the error below:  I think the issue is from Numpy but I don't know how to fix this. Please help. This is my environment:  MacOS 10.15.7 Python 3.9.0 pip 20.2.4 from /usr/local/lib/python3.9/site-packages/pip (python 3.9) Numpy 1.19.2 opencv-python 4.4.0.44 

Answer: I had the same issue when I tried to run a project I made on Windows on Mac OS. The solution I found was to install an older version of numpy (e.g. numpy 1.18). Here is the command I ran to do so : sudo python -m pip install numpy==1.18 --force  I don't think it is a good solution but it is ok for a temporary fix.
Question: I am using PYsimpleGUI in my python code, and while using the window element to create the main window, this is my code. My code: import PySimpleGUI as sg layout = [ [sg.Button('Close')] ] window = sg.Window('This is a long heading.', layout) while True: event, values = window.read() if event == sg.WIN_CLOSED or event == 'Close': break break window.close()  I notice that when I run this program, the full heading is not shown as the window resizes itself and becomes small. Is there a way to resize sg.window?

Answer: You can add size argument in the sg.Window. Try this : import PySimpleGUI as sg layout = [ [sg.Button('Close')] ] window = sg.Window('This is a long heading.', layout,size=(290, 50)) while True: event, values = window.read() if event == sg.WIN_CLOSED or event == 'Close': break break window.close()
Question: I currently have this: gsutil ls gs://basty/*_TZ001.* gs://basty/20201007_TZ001.csv gs://basty/20201008_TZ001.csv gs://basty/20201009_TZ001.csv  My problem is that I have bcuket with many files I want to delete all except 1 (20201009_TZ001.csv) I thought using bash or python I don't know.

Answer: You can filter results with grep (using -v flag to invert results) and the pipe with xargs gsutil ls gs://basty/*_TZ001.* |\ grep -v 20201009_TZ001.csv |\ xargs -i{} gsutil rm {}  To be sure that is precisely what you want, you could first execute a dry-run command: gsutil ls gs://basty/*_TZ001.* |\ grep -v 20201009_TZ001.csv |\ xargs -i{} echo "Will delete: " {}
Question: I'm currently working on an api where they send me a str range in this format: "1-5,10-25,27-30" and i need add or remove number conserving the format. if they send me "1-5,10-25,27-30" and I remove "15" the result must be "1-5,10-14,16-25,27-30" and if they send me "1-5,10-25,27-30" and i add "26" the result must be "1-5,10-30" i've been trying converting the entire range into a list of numbers, delete the target and converting it again but it's very slow doing in this way becuase they send 8-digits numbers so iter then it's not the best way how can i do this? is a library for work with this format? thanks!

Answer: intspan deals with ranges of integers and operations on them >>> from intspan import intspan >>> s = "1-5,10-25,27-30" >>> span = intspan(s) >>> str(span) '1-5,10-25,27-30' >>> span.add(26) >>> str(span) '1-5,10-30' >>> span.discard(15) >>> str(span) '1-5,10-14,16-30'
Question: My path: '/home//user////document/test.jpg'  I want this to be converted into: '/home/user/document/test.jpg'  How to do this?

Answer: Use os.path.abspath or normpath to canonicalise the path: >>> import os.path >>> os.path.abspath('/home//user////document/test.jpg') '/home/user/document/test.jpg'
Question: I want to change the variable/label names in plotly express in python. I first create a plot: import pandas as pd import plotly.express as px d = {'col1': [1, 2, 3], 'col2': [3, 4, 5]} df = pd.DataFrame(data=d) fig = px.line(df, x=df.index, y=['col1', 'col2']) fig.show()  Which yields:  I want to change the label names from col1 to hello and from col2 to hi. I have tried using labels in the figure, but I cannot get it to work: fig = px.line(df, x=df.index, y=['col1', 'col2'], labels={'col1': "hello", 'col2': "hi"}) fig.show()  But this seems to do nothing, while not producing an error. Obviously I could achieve my goals by changing the column names, but the actual plot i'm trying to create doesn't really allow for that since it comes from several different dataframes.

Answer: The answer: Without changing the data source, a complete replacement of names both in the legend, legendgroup and hovertemplate will require: newnames = {'col1':'hello', 'col2': 'hi'} fig.for_each_trace(lambda t: t.update(name = newnames[t.name], legendgroup = newnames[t.name], hovertemplate = t.hovertemplate.replace(t.name, newnames[t.name]) ) )  Plot:  The details: Using fig.for_each_trace(lambda t: t.update(name = newnames[t.name]))  ...you can change the names in the legend without changing the source by using a dict newnames = {'col1':'hello', 'col2': 'hi'}  ...and map new names to the existing col1 and col2 in the following part of the figure structure (for your first trace, col1): {'hovertemplate': 'variable=col1<br>index=%{x}<br>value=%{y}<extra></extra>', 'legendgroup': 'col1', 'line': {'color': '#636efa', 'dash': 'solid'}, 'mode': 'lines', 'name': 'hello', # <============================= here! 'orientation': 'v', 'showlegend': True, 'type': 'scatter', 'x': array([0, 1, 2], dtype=int64), 'xaxis': 'x', 'y': array([1, 2, 3], dtype=int64), 'yaxis': 'y'},  But as you can see, this doesn't do anything with 'legendgroup': 'col1', nor 'hovertemplate': 'variable=col1<br>index=%{x}<br>value=%{y}<extra></extra>' And depending on the complexity of your figure, this can pose a problem. So I would add  legendgroup = newnames[t.name] and hovertemplate = t.hovertemplate.replace(t.name, newnames[t.name])into the mix. Complete code: import pandas as pd import plotly.express as px from itertools import cycle d = {'col1': [1, 2, 3], 'col2': [3, 4, 5]} df = pd.DataFrame(data=d) fig = px.line(df, x=df.index, y=['col1', 'col2']) newnames = {'col1':'hello', 'col2': 'hi'} fig.for_each_trace(lambda t: t.update(name = newnames[t.name], legendgroup = newnames[t.name], hovertemplate = t.hovertemplate.replace(t.name, newnames[t.name]) ) )
Question: I am running a long ETL pipeline in pandas. I have to create different pandas dataframes and I want to release memory for some of the dataframes. I have been reading how to release memory and I saw that runing this command doesn't release the memory: del dataframe  Following this link: How to delete multiple pandas (python) dataframes from memory to save RAM?, one of the answer say that del statement does not delete an instance, it merely deletes a name. In the answer they say about put the dataframe in a list and then del the list: lst = [pd.DataFrame(), pd.DataFrame(), pd.DataFrame()] del lst  If I only want to release one dataframe I need to put it in a list and then delete a list like this: lst = [pd.DataFrame()] del lst  I have seen also this question: How do I release memory used by a pandas dataframe? There are different answers like: import gc del df_1 gc.collect()  Or just at the end of the dataframe use df = ""  or there is a better way to achieve that?

Answer: From the original link that you included, you have to include variable in the list, delete the variable and then delete the list. If you just add to the list, it won't delete the original dataframe, when you delete the list. import pandas import psutil import gc psutil.virtual_memory().available * 100 / psutil.virtual_memory().total >> 68.44267845153809 df = pd.read_csv('pythonSRC/bigFile.txt',sep='|') len(df) >> 20082056 psutil.virtual_memory().available * 100 / psutil.virtual_memory().total >> 56.380510330200195 lst = [df] del lst psutil.virtual_memory().available * 100 / psutil.virtual_memory().total >> 56.22601509094238 lst = [df] del df del lst psutil.virtual_memory().available * 100 / psutil.virtual_memory().total >> 76.77617073059082 gc.collect() >> 0  I tried also just deleting the dataframe and using gc.collect() with the same result! del df gc.collect() psutil.virtual_memory().available * 100 / psutil.virtual_memory().total >> 76.59363746643066  However, the execution time of adding the dataframe to the list and deleting the list and the variable is a bit faster then calling gc.collect(). I used  time.time() to measure the difference and gc.collect() was almost a full second slower! EDIT: according to the correct comment below, del df and del [df] indeed generate the same code. The problem with the original post, and my original answer is that as soon as you give a name to the list as in lst=[df], you are no longer referencing the original dataframe. lst=[df] del lst  is not the same as: del [df]
Question: I am uploading text and videos to a site by Python program. This site says they only receive video files of up to 50 MB in size. Otherwise, they will reject the video and other associated information. To ensure I can send video continuously, I want to compress it to target size (e.g. 50 MB) before sending. Because no loss of quality is impossible, it is ok to have moderate clarity loss in video or audio. Is there any elegant way in Python for this purpose? Thanks!

Answer: Compress video files by Python and FFmpeg Tools FFmpeg is a powerful tool for video editing. And there is a great Python binding named ffmpeg-python (API Reference) for this. Firstly, pip install ffmpeg-python and install FFmpeg. Steps Probe the configuration of video by function ffmpeg.probe() to get duration, audio & video bit rate and so on. And calculate the bit rate of the target file based on what we have. Then, construct commands by ffmpeg.input() and ffmpeg.output(). Finally, run it. Codes Following is the example code. Change the compression algo for your situation if you want. For easy follow-up, I hided the code of boundary condition. The code I am using is in GitHub Gist. Any bug report is welcomed! import os, ffmpeg def compress_video(video_full_path, output_file_name, target_size): # Reference: https://en.wikipedia.org/wiki/Bit_rate#Encoding_bit_rate min_audio_bitrate = 32000 max_audio_bitrate = 256000 probe = ffmpeg.probe(video_full_path) # Video duration, in s. duration = float(probe['format']['duration']) # Audio bitrate, in bps. audio_bitrate = float(next((s for s in probe['streams'] if s['codec_type'] == 'audio'), None)['bit_rate']) # Target total bitrate, in bps. target_total_bitrate = (target_size * 1024 * 8) / (1.073741824 * duration) # Target audio bitrate, in bps if 10 * audio_bitrate > target_total_bitrate: audio_bitrate = target_total_bitrate / 10 if audio_bitrate < min_audio_bitrate < target_total_bitrate: audio_bitrate = min_audio_bitrate elif audio_bitrate > max_audio_bitrate: audio_bitrate = max_audio_bitrate # Target video bitrate, in bps. video_bitrate = target_total_bitrate - audio_bitrate i = ffmpeg.input(video_full_path) ffmpeg.output(i, os.devnull, **{'c:v': 'libx264', 'b:v': video_bitrate, 'pass': 1, 'f': 'mp4'} ).overwrite_output().run() ffmpeg.output(i, output_file_name, **{'c:v': 'libx264', 'b:v': video_bitrate, 'pass': 2, 'c:a': 'aac', 'b:a': audio_bitrate} ).overwrite_output().run() # Compress input.mp4 to 50MB and save as output.mp4 compress_video('input.mp4', 'output.mp4', 50 * 1000)  Notes  Don't waste your time! Judge the file size before compressing.  You can disable two-pass function by only keeping second ffmpeg.output() without parameter 'pass': 2.  If video bit rate < 1000, it will throw exception Bitrate is extremely low.  The biggest min file size I recommend is:   # Best min size, in kB. best_min_size = (32000 + 100000) * (1.073741824 * duration) / (8 * 1024)   If you specify a extremely small target file size, the size of generated file maybe exceed it. For most time, this will not happen.
Question: I know that abs() can be used to convert numbers to positive, but is there somthing that does the opposite? I have an array full of numbers which I need to convert to negative: array1 = [] arrayLength = 25 for i in range(arrayLength): array1.append(random.randint(0, arrayLength)  I thought perhaps I could convert the numbers as they're being added, not after the array is finished. Anyone knows the code for that? Many thanks in advance

Answer: If you want to force a number to negative, regardless of whether it's initially positive or negative, you can use:  -abs(n)  Note that integer 0 will remain 0.
Question: I am tying to bulk insert a dataframe to my postgres dB. Some columns in my dataframe are date types with NaT as a null value. Which is not supported by PostgreSQL, I've tried to replace NaT (using pandas) with other NULL type identifies but that did not work during my inserts. I used df = df.where(pd.notnull(df), 'None') to replace all the NaTs, Example of errors that keep coming up due to datatype issues. Error: invalid input syntax for type date: "None" LINE 1: ...0,1.68757,'2022-11-30T00:29:59.679000'::timestamp,'None','20...  My driver and insert statement to postgresql dB: def execute_values(conn, df, table): """ Using psycopg2.extras.execute_values() to insert the dataframe """ # Create a list of tupples from the dataframe values tuples = [tuple(x) for x in df.to_numpy()] # Comma-separated dataframe columns cols = ','.join(list(df.columns)) # SQL quert to execute query = "INSERT INTO %s(%s) VALUES %%s" % (table, cols) cursor = conn.cursor() try: extras.execute_values(cursor, query, tuples) conn.commit() except (Exception, psycopg2.DatabaseError) as error: print("Error: %s" % error) conn.rollback() cursor.close() return 1 print("execute_values() done") cursor.close()  Info about my dataframe: for this case the culprits are the datetime columns only.  how is this commonly solved?

Answer: You're re-inventing the wheel. Just use pandas' to_sql method and it will  match up the column names, and take care of the NaT values.  Use method="multi" to give you the same effect as psycopg2's execute_values. from pprint import pprint import pandas as pd import sqlalchemy as sa table_name = "so64435497" engine = sa.create_engine("postgresql://scott:tiger@192.168.0.199/test") with engine.begin() as conn: # set up test environment conn.exec_driver_sql(f"DROP TABLE IF EXISTS {table_name}") conn.exec_driver_sql( f"CREATE TABLE {table_name} (" "id integer PRIMARY KEY GENERATED ALWAYS AS IDENTITY, " "txt varchar(50), " "txt2 varchar(50), " "dt timestamp)" ) df = pd.read_csv(r"C:\Users\Gord\Desktop\so64435497.csv") df["dt"] = pd.to_datetime(df["dt"]) print(df) """console output: dt txt2 txt 0 2020-01-01 00:00:00 foo2 foo 1 NaT bar2 bar 2 2020-01-02 03:04:05 baz2 baz """ # run test df.to_sql( table_name, conn, index=False, if_exists="append", method="multi" ) pprint( conn.exec_driver_sql( f"SELECT id, txt, txt2, dt FROM {table_name}" ).all() ) """console output: [(1, 'foo', 'foo2', datetime.datetime(2020, 1, 1, 0, 0)), (2, 'baz', 'baz2', None), (3, 'bar', 'bar2', datetime.datetime(2020, 1, 2, 3, 4, 5))] """
Question: I've been searching online for quite a while now and can't seem to find a solution for my problem. I installed Pylance (the newest Microsoft interpreter for Python) and can't seem to disable linting at all. I've tried a lot of options but none worked. Here's a screenshot of how annoying linting is in my code now. Here's how my VSCode Settings file looks like: { // "python.pythonPath": "C://Anaconda3//envs//py34//python.exe", // "python.pythonPath": "C://Anaconda3_2020//python.exe", // "python.pythonPath": "C://Anaconda3_2020_07//python.exe", "python.pythonPath": "C://Anaconda3//python.exe", "python.analysis.disabled": [ "unresolved-import" ], "editor.suggestSelection": "first", "editor.fontSize": 15, "typescript.tsserver.useSeparateSyntaxServer": false, "workbench.colorTheme": "Monokai ST3", "workbench.colorCustomizations": { "editor.background": "#000000", "statusBar.background": "#000000", "statusBar.noFolderBackground": "#212121", "statusBar.debuggingBackground": "#263238" }, "window.zoomLevel": 0, "editor.renderLineHighlight": "none", "editor.fontFamily": "Meslo LG L", "editor.tabCompletion": "on", "editor.parameterHints.enabled": true, "python.terminal.executeInFileDir": true, "python.terminal.launchArgs": [ "-u" ], "terminal.integrated.shell.windows": "C:\\Windows\\System32\\WindowsPowerShell\\v1.0\\powershell.exe", "editor.lineHeight": 0, "workbench.editor.scrollToSwitchTabs": true, "python.autoComplete.showAdvancedMembers": false, "python.languageServer": "Pylance", "python.linting.enabled": false, "python.linting.pylintEnabled": false, "python.linting.lintOnSave": false, "python.linting.flake8Enabled": false, "python.linting.mypyEnabled": false, "python.linting.banditEnabled": false, "python.linting.pylamaEnabled": false, "python.linting.pylintArgs": [ "--unsafe-load-any-extension=y", "--load-plugin", "pylint_protobuf", "--disable=all", "--disable=undefined-variable", ], "python.linting.mypyArgs": [ "--ignore-missing-imports", "--follow-imports=silent", "--show-column-numbers", "--extension-pkg-whitelist=all", "--disable=all", "--disable=undefined-variable", ], }   Any thoughts? Any help is much appreciated.

Answer: You can disable the language server with: "python.languageServer": "None"