eshangj/stackoverflow_q_and_a_sample huggingface dataset:
Question: I have Python project and have installed hundreds of packages using pip over time. However, I have since improved my code and suspect that some of these packages are no longer in use. I want to remove all of the unused packages. I am already familiar with the pip uninstall package-name command and have seen suggestions to use pip-autoremove package-name, but it's not considered to be the optimal option for my case as there are hundreds of unused packages, and I do not want to manually uninstall each one. Additionally, I do not want to accidentally remove packages that are still being used in my project. I am wondering if there is a better solution, or a way to look for python imports inside files and detect for unused. Any latest or alternative approaches would be highly appreciated. Thank you. Here's my requirements.txt with thousands of packages: 

Answer: Dependency management in Python is a painful topic. Check out this excellent overview: https://chriswarrick.com/blog/2023/01/15/how-to-improve-python-packaging/ I highly recommend you check out Poetry or Hatch for dependency and packaging management. In general you should at least always use virtual environment. This allows you to completely delete it and create new, if you need. You have your requirements.txt so you can do python -m venv <venv> where <venv> is the name you want to give to your virtual environment. It creates a directory where you will install the packages. Activating the virtual environment goes like this: source <venv>/bin/activate which executes a shell script. Among other things it defines a command deactivate which you can call if you wish to exit from the virtual environment. With the activated venv everything you install will be installed there.
Question: I have a Spark data frame (df1) with a particular schema, and I have another dataframe with the same columns, but different schema. I know how to do it column by column, but since I have a large set of columns, it would be quite lengthy. To keep the schema consistent across dataframes, I was wondering if I could be able to apply one schema to another data frame or creating a function that do the job. Here is an example: df1 # root # |-- A: date (nullable = true) # |-- B: integer (nullable = true) # |-- C: string (nullable = true) df2 # root # |-- A: string (nullable = true) # |-- B: string (nullable = true) # |-- C: string (nullable = true)`  I want to copy apply the schema of df1 to df2. I tried this approach for one column. Given that I have a large number of columns, it would be quite a lengthy way to do it. df2 = df2.withColumn("B", df2["B"].cast('int')) 

Answer: Yes, its possible dynamically with dataframe.schema.fields df2.select(*[(col(x.name).cast(x.dataType)) for x in df1.schema.fields]) Example: from pyspark.sql.functions import * df1 = spark.createDataFrame([('2022-02-02',2,'a')],['A','B','C']).withColumn("A",to_date(col("A"))) print("df1 Schema") df1.printSchema() #df1 Schema #root # |-- A: date (nullable = true) # |-- B: long (nullable = true) # |-- C: string (nullable = true) df2 = spark.createDataFrame([('2022-02-02','2','a')],['A','B','C']) print("df2 Schema") df2.printSchema() #df2 Schema #root # |-- A: string (nullable = true) # |-- B: string (nullable = true) # |-- C: string (nullable = true) # #casting the df2 columns by getting df1 schema using select clause df3 = df2.select(*[(col(x.name).cast(x.dataType)) for x in df1.schema.fields]) df3.show(10,False) print("df3 Schema") df3.printSchema() #+----------+---+---+ #|A |B |C | #+----------+---+---+ #|2022-02-02|2 |a | #+----------+---+---+ #df3 Schema #root # |-- A: date (nullable = true) # |-- B: long (nullable = true) # |-- C: string (nullable = true)  In this example I have df1 defined with Integer,date,long types. df2 is defined with string type. df3 is defined by using df2 as source data and attached df1 schema.
Question: I know how to align annotation respective to the plotting area (using xref="paper"). I want to place the annotation to right bottom corner of the whole figure. Absolute or relative offset does not work, since the width of the margin outside the plotting area is changing among several figures, depending on their content. Is there a way how to do it? Thanks a ton. Edit: Example code import plotly.express as px fig = px.scatter(x=[1, 2, 3], y=[1, 2, 3], title="Try panning or zooming!") fig.add_annotation(text="Absolutely-positioned annotation", xref="paper", yref="paper", x=0.3, y=0.3, showarrow=False) fig.show()  This annotation is positioned respectively to the origin of the plot, not respectively to the plot edges. 

Answer: So the thing is that you technically can't automatically calculate and position annotations at a perfect spot in the corner of the figure for every plot (outside of the plot, but inside the figure). The problem is that to do so, you would need to be able to get the dimensions of every plot, which could be undefined when you do not specify a width or height (autosizing will automatically calculate dimensions when the plot is shown). And even if you could do this, when you resize the window/page the annotation's location will change, and no longer be in the corner as you wanted it to be. Refer here for a proper explanation. However, you could work around this by embedding the plotly plot inside an app/frame, and add the text to that frame instead of the plot itself. Here is a solution to a similar question asked by someone else from before that uses the jinja2 module's Template object to embed the plot (though it creates HTML code which you will then need to separately render). Using the same method, you could implement a frame into the example plot like so: import plotly.express as px from jinja2 import Template fig = px.scatter(x=[1, 2, 3], y=[1, 2, 3], title="Try panning or zooming!") # Your annotation code and figure showing ''' fig.add_annotation(text="Absolutely-positioned annotation", xref="paper", yref="paper", x=1, y=-0.1, showarrow=False) fig.show() ''' version_number = 'Absolutely-positioned annotation' #Your text to be displayed fig_html = {"fig":fig.to_html(full_html=False, include_plotlyjs='cdn')} #Convert your plot to HTML to embed # HTML Frame to embed the figure and text frame = """<!DOCTYPE html> <html> <body> {{ fig }} <p style="font-size:15px; position:fixed; bottom:0; right:0;">{{ version_number }}</p> </body> </html>""" j2_template = Template(frame) #Create the Template object for our frame plot = j2_template.render(fig_html,version_number=version_number) #Embed the plot and text into the frame  At this point you have yourself HTML code with your plot and the text you desire. To render it, use any module capable of rendering HTML code, or write it to an HTML file and open that instead. Example: from flask import Flask, render_template_string app = Flask(__name__) @app.route('/') def home(): return render_template_string(plot) #variable containing HTML from previous code if __name__ == '__main__': app.run(debug=True)
Question: Does anyone know why polars (or maybe my pycharm setup or python debugger) limits the number of rows in the output? This drives me nuts. Here is the polars code i am running but I do suspect its not polars specific as there isnt much out there on google (and chatgpt said its info is too old haha). import polars as pl df = pl.scan_parquet('/path/to/file.parquet') result_df =( df .filter(pl.col("condition_category") == 'unknown') .groupby("type") .agg( [ pl.col("type").count().alias("counts"), ] ) ).collect() print(result_df)  

Answer: Looks like the following will work. Thanks to @wayoshi for sharing this. I will say that the defaults are way too conservative! with pl.Config(tbl_rows=1000): print(result_df)  or throw this at the top of your script if you prefer to not manage contexts. import polars as pl # Configure Polars cfg = pl.Config() cfg.set_tbl_rows(2000)
Question: So what is happening is the values in column B are becoming NaN. How would I fix this so that it does not override other values? import pandas as pd import numpy as np # %% # df=pd.read_csv('testing/example.csv') data = { 'Name' : ['Abby', 'Bob', 'Chris'], 'Active' : ['Y', 'Y', 'N'], 'A' : [89, 92, np.nan], 'B' : ['eye', 'hand', np.nan], 'C' : ['right', 'left', 'right'] } df = pd.DataFrame(data) df.loc[((df['Active'] =='N') & (df['A'].isna())), ['A', 'B']] = [99, df['C']] df  What I want the results to be is:     Name Active A B C     Abby Y 89.0 eye right   Bob Y 92.0 hand left   Chris N 99 right right    

Answer: You can implement this by creating a boolean mask using the condition: Where the 'Active' column is 'N' and the 'A' column has missing values (np.nan). We can then use this mask to filter rows in the DataFrame. First, we will replace the values in column 'A' with 99 for rows where the mask condition is True. Then, replace the values in column 'B' with the corresponding values from column 'C' where the mask condition is True. Following is the modified code: import pandas as pd import numpy as np data = { 'Name' : ['Abby', 'Bob', 'Chris'], 'Active' : ['Y', 'Y', 'N'], 'A' : [89, 92, np.nan], 'B' : ['eye', 'hand', np.nan], 'C' : ['right', 'left', 'right'] } df = pd.DataFrame(data) mask = (df['Active'] =='N') & (df['A'].isna()) df.loc[mask, 'A'] = 99 df.loc[mask, 'B'] = df.loc[mask, 'C'] print(df)  Now, the DataFrame will be updated correctly. Following is the output:  Name Active A B C 0 Abby Y 89.0 eye right 1 Bob Y 92.0 hand left 2 Chris N 99.0 right right
Question: The following message means that Python has not been installed completely. If Yes! Do I have to install the 'lzma' extension? ModuleNotFoundError: No module name '_lzma'  Warning: The Python lzma extension was not compiled. Missing the lzma lib?  Installed Python-3.11.3 to /Users/admin/.pyenv/versions/3.11.3 Thank you!

Answer: Per Chris' comment, the solution is to install OS-specific dependencies then use pyenv to install a python version.
Question: When applying isort 5.12.0 in pre-commit within a python file, it re-orders the imports in bad. In bitbucket pipelines, the same code orders correctly. This correct code: from dagster import build_init_resource_context from module1 import setting from module1.resources.apple import AppleConnector as apple_connector from module1.resources.apple import apple_resource from module1.samples.apple import ( apple_schema as apple_schemas, ) from jsonschema import validate  Gets re-ordered this way: from dagster import build_init_resource_context from module1.resources.apple import AppleConnector as apple_connector from module1.resources.apple import apple_resource from module1.samples.apple import ( apple_schema as apple_schemas, ) from jsonschema import validate from module1 import setting  ¿Why is it happening?

Answer: In .pre-commit-config.yaml isort configuration, I had to add module1 as a first-party package. That avoided different interpretations in different repositories. Then pre-commit worked consistently in all the pipelines. - repo: https://github.com/PyCQA/isort rev: 5.12.0 hooks: - id: isort args: ["--profile", "black", "--filter-files", "--project", "module1"]
Question: If a program exits before all of its input has been consumed, then the remaining input will be sent to the shell. Here's an example: import sys for line in sys.stdin: sys.exit()  Try running the example and copy-paste this multi-line input: foo bar baz  The result will look like this: ❯ python example.py foo bar baz% ❯ bar zsh: command not found: bar ❯ baz  In this case, the program exited after consuming foo, so bar was automatically sent to the shell. How can I clear the remaining input before Python exits? The for loop in the example represents complicated logic, so I'm looking for a solution that doesn't modify the for loop. I tried registering an atexit handler to clear the input: import sys, atexit def clear_stdin(): sys.stdin.read() atexit.register(clear_stdin) for line in sys.stdin: sys.exit()  This solution does clear the input and prevent it from being sent to the shell, but it unfortunately causes the program to hang until the user enters a blank line. I'd like to clear the remaining input without pausing the program.

Answer: On Unix systems you can use os.set_blocking before reading the input. If the user pastes a large amount of text, then the terminal may be holding on to additional input that hasn't been sent to stdin yet. It may not be possible for the program to see this, so the following code includes a sleep to allow for the terminal to provide more input. This code also includes several guards to prevent errors: import sys, atexit, os from time import sleep def clear_stdin(): if sys.__stdin__ and sys.__stdin__.isatty() and hasattr(os, 'set_blocking'): try: os.set_blocking(sys.__stdin__.fileno(), False) except OSError: return try: while sys.__stdin__.read(): sleep(0.1) except TypeError: return atexit.register(clear_stdin) for line in sys.stdin: sys.exit()  This will prevent the remaining input from being sent to the shell while still allowing the program to exit immediately: ❯ python example.py foo bar baz% ❯ baz  If the last line does not end with a newline character then it won't be consumed, but it won't be executed by the shell either.
Question: I tried to upgrade conda and spyder, but I encountered an issue. I tried:  conda update anaconda conda install spyder=5.4.3 pip install pylsp-mypy  I used option 1 in the spyder terminal first and then I used it in the conda prompt, I don't know if that caused the error. How can I solve this issue?  Edit (temporal solution): I used conda install --revision 0, and this fresh Anaconda installation, with the drawback of needing to install every package again. However, all environments created remain intact. I don't know what happened, but when I write 1 and 2 in the prompt again, I got the same error. I don't know if is a version issue.

Answer: Please use this instead: conda install -c conda-forge python-lsp-server
Question: I'm working with auto-gpt and I got this error: Command evaluate_code returned: Error: The model: `gpt-4` does not exist  and it's like it can't go furthermore. what should I do?

Answer: Update your .env file in your repo so that both are = to gpt-3.5-turbo SMART_LLM_MODEL=gpt-3.5-turbo FAST_LLM_MODEL=gpt-3.5-turbo  this issue is happening because you do not have API access to GPT4.
Question: I am using pandas to check wether two dataframes are contained within each others. the method .isin() is only helpful (e.g., returns True) only when labels match (ref: https://pandas.pydata.org/docs/reference/api/pandas.DataFrame.isin.html) but I want to check further that this to include cases where the labels don't match. Example: df1: +----+----+----+----+----+ | 3 | 4 | 5 | 6 | 7 | +----+----+----+----+----+ | 11 | 13 | 10 | 15 | 12 | +----+----+----+----+----+ | 8 | 2 | 9 | 0 | 1 | +----+----+----+----+----+ | 14 | 23 | 31 | 21 | 19 | +----+----+----+----+----+  df2: +----+----+ | 13 | 10 | +----+----+ | 2 | 9 | +----+----+  I want the output to be True since df2 is inside df1 Any ideas how to do that using Pandas?

Answer: You can use numpy's sliding_window_view: from numpy.lib.stride_tricks import sliding_window_view as swv (swv(df1, df2.shape)==df2.to_numpy()).all((-2, -1)).any()  Output: True Intermediate: (swv(df1, df2.shape)==df2.to_numpy()).all((-2, -1)) array([[False, False, False, False], [False, True, False, False], # df2 is found in position 1,1 [False, False, False, False]])  using a partial match Example 1: ≥ 75% of matches: from numpy.lib.stride_tricks import sliding_window_view as swv ((swv(df1, df2.shape)==df2.to_numpy()).mean((-2, -1))>=0.75).any()  Example 2: ≥ 3 matches: from numpy.lib.stride_tricks import sliding_window_view as swv ((swv(df1, df2.shape)==df2.to_numpy()).sum((-2, -1))>=3).any()  Alternative input: df2 = pd.DataFrame([[13, 10], [2, 8]])
Question: I have built a web application using streamlit and hosted it on the Google Cloud Platform (App Engine). The URL is something like https://xxx-11111.uc.r.appspot.com/ which is given for the Stream URL. I enabled Google Analytics 2 days back but apparently, it is not set up correctly. It was given that I need to add in the head tag. This is the code where I added the Google Analytics tag... What is wrong?? def page_header(): st.set_page_config(page_title="xx", page_icon="images/logo.png") header = st.container() with header: # Add banner image logo = Image.open("images/logo.png") st.image(logo, width=300) # Add Google Analytics code to the header ga_code = """ <!-- Google tag (gtag.js) --> <script async src="https://www.googletagmanager.com/gtag/js?id=G-xxxxxx"></script> <script> window.dataLayer = window.dataLayer || []; function gtag(){dataLayer.push(arguments);} gtag('js', new Date()); gtag('config', 'G-xxxxxx'); </script> """ st.markdown(ga_code, unsafe_allow_html=True) # Define the main function to run the app def main(): # Render the page header page_header() ..... if __name__ == "__main__": main() 

Answer: One way to implement Google Analytics into your GAE Streamlit app would be to add the GA global site tag JS code to the default /site-packages/streamlit/static/index.html file NOTE: This script should be run prior to running the streamlit server from bs4 import BeautifulSoup import shutil import pathlib import logging import streamlit as st def add_analytics_tag(): # replace G-XXXXXXXXXX to your web app's ID analytics_js = """ <!-- Global site tag (gtag.js) - Google Analytics --> <script async src="https://www.googletagmanager.com/gtag/js?id=G-XXXXXXXXXX"></script> <script> window.dataLayer = window.dataLayer || []; function gtag(){dataLayer.push(arguments);} gtag('js', new Date()); gtag('config', 'G-XXXXXXXXXX'); </script> <div id="G-XXXXXXXXXX"></div> """ analytics_id = "G-XXXXXXXXXX" # Identify html path of streamlit index_path = pathlib.Path(st.__file__).parent / "static" / "index.html" logging.info(f'editing {index_path}') soup = BeautifulSoup(index_path.read_text(), features="html.parser") if not soup.find(id=analytics_id): # if id not found within html file bck_index = index_path.with_suffix('.bck') if bck_index.exists(): shutil.copy(bck_index, index_path) # backup recovery else: shutil.copy(index_path, bck_index) # save backup html = str(soup) new_html = html.replace('<head>', '<head>\n' + analytics_js) index_path.write_text(new_html) # insert analytics tag at top of head
Question: I have a Linux Server. On that I have two Docker Containers.In the first one I am deploying my Flutter Web and in the other one I am running my RestAPI with FastAPI(). I set both the Docker containers in the same Network, so the communication should work. I also set origins with origins = ['*'] (Wildcard). I reverse proxy my Flutter web with nginx from the Linux server. I also include *.crt and *.key with nginx to my Flutter Web. Now, obviously, since my Flutter Web App has https, I cant make http calls. When I am trying to make a call with https, I get the Error (from catch): "XMLHttpRequest error", and in the Browser Console I get:  Cross-Origin Request Blocked: The Same Origin Policy disallows reading the remote resource at https://172.21.0.2:8070/. (Reason: CORS request did not succeed). Status code: (null).  (172.21.0.2 is the ip of the Docker and 8070 the port on RestApi running) I am new to the RestAPI world. I normaly develop only Frontend. But I wanted to give it a try. So I'm sorry if I expressed some things wrong. I am searching since days but cant find a solution to my problem. I would be grateful for any help! (If i missed some information or you need more, feel free to write in the comments, I will update the Question immediately!) Thank You!

Answer: I solved my problem. If anyone face the same problem in the future, here is the answer. My mistake was to call the server with http/https, while in the same (Docker) Network. So I changed: final Uri tokenUri = Uri.https(urlList[index]['url']!, '');  to final Uri tokenUri = Uri.parse('${urlList[index]['url']!}/');  One week of search for such an easy solution.
Question: I’ve trained a custom tokenizer using a custom dataset using this code that’s on the documentation. Is there a method for me to add this tokenizer to the hub and to use it as the other tokenizers by calling the AutoTokenizer.from_pretrained() function? If I can’t do that how can I use the tokenizer to train a custom model from scratch? Thanks for your help!!! Here's the code below: from tokenizers import Tokenizer from tokenizers.models import BPE tokenizer = Tokenizer(BPE(unk_token="[UNK]")) from tokenizers.trainers import BpeTrainer trainer = BpeTrainer(special_tokens=["[UNK]", "[CLS]", "[SEP]", "[PAD]", "[MASK]"]) from tokenizers.pre_tokenizers import Whitespace tokenizer.pre_tokenizer = Whitespace() folder = 'dataset_unicode' files = [f"/content/drive/MyDrive/{folder}/{split}.txt" for split in ["test", "train", "valid"]] tokenizer.train(files, trainer) from tokenizers.processors import TemplateProcessing tokenizer.post_processor = TemplateProcessing( single="[CLS] $A [SEP]", pair="[CLS] $A [SEP] $B:1 [SEP]:1", special_tokens=[ ("[CLS]", tokenizer.token_to_id("[CLS]")), ("[SEP]", tokenizer.token_to_id("[SEP]")), ], ) # I've tried saving it like this but it doesn't work as I expect it: tokenizer.save("data/tokenizer-custom.json") 

Answer: The AutoTokenizer expects a few files in the directory: awesometokenizer/ tokenizer_config.json special_tokens_map.json tokenizer.json  But the default tokenizer.Tokenizer.save() function only saves the vocab file in awesometokenizer/tokenizer.json, open up the json file and compare the ['model']['vocab'] keys to your json from data/tokenizer-custom.json. The simplest way to let AutoTokenizer load .from_pretrained is to follow the answer that @cronoik posted in the comment, using PreTrainedTokenizerFast, i.e. adding a few lines to your existing code: from tokenizers import Tokenizer from tokenizers.models import BPE from tokenizers.trainers import BpeTrainer from tokenizers.pre_tokenizers import Whitespace from tokenizers.processors import TemplateProcessing from transformers import PreTrainedTokenizerFast # <---- Add this line. trainer = BpeTrainer(special_tokens=["[UNK]", "[CLS]", "[SEP]", "[PAD]", "[MASK]"]) tokenizer = Tokenizer(BPE(unk_token="[UNK]")) tokenizer.pre_tokenizer = Whitespace() files = ["big.txt"] # e.g. training with https://norvig.com/big.txt tokenizer.train(files, trainer) tokenizer.post_processor = TemplateProcessing( single="[CLS] $A [SEP]", pair="[CLS] $A [SEP] $B:1 [SEP]:1", special_tokens=[ ("[CLS]", tokenizer.token_to_id("[CLS]")), ("[SEP]", tokenizer.token_to_id("[SEP]")), ], ) # Add these lines: # | # | # V awesome_tokenizer = PreTrainedTokenizerFast(tokenizer_object=tokenizer) awesome_tokenizer.save_pretrained("awesome_tokenizer")  Then you can load the trained tokenizer: from transformers import AutoTokenizer auto_loaded_tokenizer = AutoTokenizer.from_pretrained( "awesome_tokenizer", local_files_only=True )  Note: tokenizers though can be pip installed, is a library in Rust with Python bindings
Question: Given a sql query such as query = """ select some_col from tbl where some_col > :value """  I'm executing this with sqlalchemy using connection.execute(sa.text(query), {'value' : 5})  Though this does what's expected, I would like to be able to get the raw sql, with replaced parameters. Meaning I would like a way to be able to get select some_column from tbl where some_column > 5  I've tried to echo the sql using:  engine = sa.create_engine( '<CONNECTION STRING>', echo=True, )  But this didn't replace the parameters. If there's not a way to do this in sqlalchemy, but is a way using something like psycopg2 (as long as the syntax :value doesn't change) then that would be of interest.

Answer: The SQLAlchemy documentation for Rendering Bound Parameters Inline explains how we can use literal_binds: query = "select some_col from tbl where some_col > :value" print( sa.text(query) .bindparams(value=5) .compile(compile_kwargs={"literal_binds": True}) ) # select some_col from tbl where some_col > 5