eshangj/stackoverflow_q_and_a_sample huggingface dataset:
Question: In pandas, one can do: import pandas as pd d = {"foo":[1,2,3, None], "bar":[4,None, None, 6]} df_pandas = pd.DataFrame.from_dict(d) dict(df_pandas.isnull().sum())  [out]: {'foo': 1, 'bar': 2}  In polars it's possible to do the same by looping through the columns: import polars as pl d = {"foo":[1,2,3, None], "bar":[4,None, None, 6]} df_polars = pl.from_dict(d) {col:df_polars[col].is_null().sum() for col in df_polars.columns}  Looping through the columns in polars is particularly painful when using LazyFrame, then the .collect() has to be done in chunks to do the aggregation. Is there a way to find no. of nulls in every column in a polars dataframe without looping through each columns?

Answer: Assuming you're not married to the output format the idiomatic way to do it is... df.select(pl.all().is_null().sum())  However if you really like the dict output you can easily get it... df.select(pl.all().is_null().sum()).to_dicts()[0]  The way this works is that inside the select we start with pl.all() which means all of the columns and then, much like in the pandas version, we apply is_null which would return True/False. From that we chain sum which turns the Trues into 1s and gives you the number of nulls in each column. There's also the dedicated null_count() so you don't have to chain is_null().sum() thanks to @jqurious for that tip.
Question: Im trying to convert a matlab script into python. The script read a binary file and reshape into a number of columns. The matlab script is: fid=fopen(binary_file,'rb'); [inpar,ic]=fread(fid,4,'int'); if (ic<4) ; idata=[];return;end nmagic=inpar(1); nh=inpar(2); nrpar=inpar(3); nipar=inpar(4); [rdata,ic]=fread(binary_file,[nh,nrpar],'float'); if (ic<nh*nrpar) ; return;end [idata,ic]=fread(fid,[nh,nipar],'int'); if (ic<nh*nipar) ; return;end  The python code I tried is: import numpy as np inpar = np.fromfile(fid, dtype=np.int32) nmagic, nh, nrpar, nipar = inpar rdata = np.fromfile(fid, dtype=np.float32, count=nh * nrpar).reshape(nh, nrpar) idata = np.fromfile(fid, dtype=np.int32, count=nh * nipar).reshape(nh, nipar)  I don't exactly know how Matlab is reshaping the data. Can anyone help me translate the code to achieve the same result in Python. A sample data is given here

Answer: Lets try to read the data with open in rb mod so it will be read in binary mode. Then we will use np.fromfile to read from the file. import numpy as np with open(binary_file, 'rb') as fid: inpar = np.fromfile(fid, dtype=np.int32, count=4) if inpar.size < 4: idata = [] else: nmagic, nh, nrpar, nipar = inpar rdata = np.fromfile(fid, dtype=np.float32, count=nh * nrpar) if rdata.size < nh * nrpar: # handle error pass else: rdata = rdata.reshape(nh, nrpar) idata = np.fromfile(fid, dtype=np.int32, count=nh * nipar) if idata.size < nh * nipar: # handle error pass else: idata = idata.reshape(nh, nipar)
Question: I have Dataframe with stuct inside after used pl.cum_fold(). How struct convert to normal series based column? ┌───────────────────────────────────┐ │ price │ │ --- │ │ struct[2000] │ ╞═══════════════════════════════════╡ │ {null,null,null,null,30302.67187… │ └───────────────────────────────────┘  Is it possible with polars expression or some kind of util method? I find the only way is convert to Python and then back to DF. df = pl.DataFrame(pl.Series('price', df[0, 0].values())) 

Answer: There is transpose once the struct is unnested. This will be relatively slow but I don't think there is any other way: df = pl.DataFrame( {"price": {'test0' : None, 'test1' : 1, 'test2' : 2}}) df.unnest('price').transpose(column_names=['price'])  shape: (3, 1) ┌───────┐ │ price │ │ --- │ │ f64 │ ╞═══════╡ │ null │ │ 1.0 │ │ 2.0 │ └───────┘
Question: I am new to polars and I wonder what is the equivalent of pandas groupby.apply(drop_duplicates) in polars. Here is the code snippet I need to translate : import pandas as pd GROUP = list('123231232121212321') OPERATION = list('AAABBABAAAABBABBBA') BATCH = list('777898897889878987') df_input = pd.DataFrame({'GROUP':GROUP, 'OPERATION':OPERATION, 'BATCH':BATCH}) df_output = df_input.groupby('GROUP').apply(lambda x: x.drop_duplicates())  input data desired output I tried the following, but, it does not output what I need import polars as pl GROUP = list('123231232121212321') OPERATION = list('AAABBABAAAABBABBBA') BATCH = list('777898897889878987') df_input = pl.DataFrame({'GROUP':GROUP, 'OPERATION':OPERATION, 'BATCH':BATCH}) df_output = df_input.group_by('GROUP').agg(pl.all().unique())  If I take only one Group, I get locally what I want : df_part = df_input.filter(pl.col('GROUP')=='2') df_part[['OPERATION', 'BATCH']].unique()  Does somebody know how to do that ?

Answer: It looks like you want the first instance of each OPERATION, BATCH "pairing" per GROUP You can use pl.struct to create the "pairing" and then use is_first_distinct() as a Window function. (df.with_row_index() .filter(pl.struct("OPERATION", "BATCH").is_first_distinct().over("GROUP")) )  shape: (9, 4) ┌───────┬───────┬───────────┬───────┐ │ index ┆ GROUP ┆ OPERATION ┆ BATCH │ │ --- ┆ --- ┆ --- ┆ --- │ │ u32 ┆ str ┆ str ┆ str │ ╞═══════╪═══════╪═══════════╪═══════╡ │ 0 ┆ 1 ┆ A ┆ 7 │ │ 1 ┆ 2 ┆ A ┆ 7 │ │ 2 ┆ 3 ┆ A ┆ 7 │ │ 3 ┆ 2 ┆ B ┆ 8 │ │ 4 ┆ 3 ┆ B ┆ 9 │ │ 5 ┆ 1 ┆ A ┆ 8 │ │ 7 ┆ 3 ┆ A ┆ 9 │ │ 10 ┆ 2 ┆ A ┆ 8 │ │ 11 ┆ 1 ┆ B ┆ 9 │ └───────┴───────┴───────────┴───────┘  The with_row_index is just used here as a visual guide to help see the removed rows. (index 8, 9)
Question: Say I have this: >>> polars.DataFrame([[(1,2),(3,4)],[(5,6),(7,8)]], list('ab')) shape: (2, 2) ┌────────┬────────┐ │ a ┆ b │ │ --- ┆ --- │ │ object ┆ object │ ╞════════╪════════╡ │ (1, 2) ┆ (5, 6) │ │ (3, 4) ┆ (7, 8) │ └────────┴────────┘  How would I go with updating the 2nd element of the tuple only? E.g. what do I need to do to get this: ┌────────┬────────┐ │ a ┆ b │ │ --- ┆ --- │ │ object ┆ object │ ╞════════╪════════╡ │ (1, 9) ┆ (5, 9) │ │ (3, 9) ┆ (7, 9) │ └────────┴────────┘  where I've replaced the 2nd element of all tuples to 9. Note that I'm not necessarily married to using tuples, but I'd like some kind of structure (dict, list, ...) as I want elements to have two pieces of information per cell due to a downstream requirement.

Answer: If you choose dicts instead, Polars will turn them into Structs df = pl.DataFrame({ "a": [{"x": 1, "y": 2}, {"x": 3, "y": 4}], "b": [{"x": 5, "y": 6}, {"x": 7, "y": 8}] })  shape: (2, 2) ┌───────────┬───────────┐ │ a ┆ b │ │ --- ┆ --- │ │ struct[2] ┆ struct[2] │ ╞═══════════╪═══════════╡ │ {1,2} ┆ {5,6} │ │ {3,4} ┆ {7,8} │ └───────────┴───────────┘  You can then set the new value easily using .with_fields() df.with_columns( pl.all().struct.with_fields(y = 9) )  shape: (2, 2) ┌───────────┬───────────┐ │ a ┆ b │ │ --- ┆ --- │ │ struct[2] ┆ struct[2] │ ╞═══════════╪═══════════╡ │ {1,9} ┆ {5,9} │ │ {3,9} ┆ {7,9} │ └───────────┴───────────┘
Question: The docs for sys.path state the following:  A list of strings that specifies the search path for modules. Initialized from the environment variable PYTHONPATH, plus an installation-dependent default.  So my understanding here is that PYTHONPATH is an environment variable. Environment variables can be printed out in Powershell using the following command: PS> echo $ENV:VARIABLENAME  However when I do $ENV:PYTHONPATH I get no output. If I try to access PYTHONPATH from a python terminal, I get a KeyError: >>> import os >>> os.environ["PYTHONPATH"] Traceback (most recent call last): File "<stdin>", line 1, in <module> File "C:\Python310\lib\os.py", line 679, in __getitem__ raise KeyError(key) from None KeyError: 'PYTHONPATH'  However, I know PYTHONPATH is defined somewhere, because its value does appear when I use sys.path: >>> import sys >>> sys.path ['', 'C:\\Python310\\python310.zip', 'C:\\Python310\\DLLs', 'C:\\Python310\\lib', 'C:\\Python310', 'C:\\Users\\aa\\AppData\\Roaming\\Python\\Python310\\site-packages', 'C:\\Python310\\lib\\site-packages', 'C:\\Python310\\lib\\site-packages\\scons-4.4.0-py3.10.egg', 'C:\\Python310\\lib\\site-packages\\colorama-0.3.2-py3.10.egg', 'C:\\Python310\\lib\\site-packages\\win32', 'C:\\Python310\\lib\\site-packages\\win32\\lib', 'C:\\Python310\\lib\\site-packages\\Pythonwin']  If PYTHONPATH is truly an environment variable, why can't I access it using either Powershell or os in my Python interpreter?

Answer: The variable PYTHONPATH is an environment variable which you can set to add additional directories where python will look for modules and packages. This variable is not set by default and not needed for Python to work because it it already knows where to find its standard libraries (sys.path). But if for any reason you need some custom Python libraries that you do not want to install, you can export PYTHONPATH=/path/to/my/modules/ to add /path/to/my/modules/ on Python will then know where to find those custom modules. And this time, if you print again sys.path, it will display you the newly added modules.
Question: After pip install openai, when I try to import openai, it shows this error:  the 'ssl' module of urllib3 is compile with LibreSSL not OpenSSL  I just followed a tutorial on a project about using API of OpenAI. But when I get to the first step which is the install and import OpenAI, I got stuck. And I tried to find the solution for this error but I found nothing. Here is the message after I try to import OpenAI: Python 3.9.6 (default, Mar 10 2023, 20:16:38) [Clang 14.0.3 (clang-1403.0.22.14.1)] on darwin Type "help", "copyright", "credits" or "license" for more information. >>> import openai Traceback (most recent call last): File "<stdin>", line 1, in <module> File "/Users/yule/Library/Python/3.9/lib/python/site-packages/openai/__init__.py", line 19, in <module> from openai.api_resources import ( File "/Users/mic/Library/Python/3.9/lib/python/site-packages/openai/api_resources/__init__.py", line 1, in <module> from openai.api_resources.audio import Audio # noqa: F401 File "/Users/mic/Library/Python/3.9/lib/python/site-packages/openai/api_resources/audio.py", line 4, in <module> from openai import api_requestor, util File "/Users/mic/Library/Python/3.9/lib/python/site-packages/openai/api_requestor.py", line 22, in <module> import requests File "/Users/mic/Library/Python/3.9/lib/python/site-packages/requests/__init__.py", line 43, in <module> import urllib3 File "/Users/mic/Library/Python/3.9/lib/python/site-packages/urllib3/__init__.py", line 38, in <module> raise ImportError( ImportError: urllib3 v2.0 only supports OpenSSL 1.1.1+, currently the 'ssl' module is compiled with LibreSSL 2.8.3. See: https://github.com/urllib3/urllib3/issues/2168  I tried to --upgrade the urllib3, but it is still not working. The result is: pip3 install --upgrade urllib3 Defaulting to user installation because normal site-packages is not writeable Requirement already satisfied: urllib3 in ./Library/Python/3.9/lib/python/site-packages (2.0.2) 

Answer: The reason why the error message mentioned OpenSSL 1.1.1+ and LibreSSL 2.8.3 is that urllib3 v2.0 (the version you've installed) requires OpenSSL 1.1.1+ to work properly, as it relies on some new features of OpenSSL 1.1.1. The issue is that the version of the 'ssl' module that is currently installed in your environment is compiled with LibreSSL 2.8.3, which is not compatible with urllib3 v2.0. To use urllib3 v2.0, you need an 'ssl' module compiled with OpenSSL 1.1.1 or later, by trying: brew install openssl@1.1  Or you could use an older version of urllib3 that is compatible suc. For example urllib3 v1.26.6, which does not have a strict OpenSSL version requirement. You can force the version installing with this command: pip install urllib3==1.26.6
Question: import tweepy import os api_key = os.getenv('TWI_API_KEY') api_key_secret = os.getenv('TWI_API_KEY_SECRET') bearer_token = os.getenv('TWI_BEARER_TOKEN') access_token = os.getenv('TWI_ACCESS_TOKEN') access_token_secret = os.getenv('TWI_ACCESS_TOKEN_SECRET') client = tweepy.Client(bearer_token=bearer_token, consumer_key=api_key, consumer_secret=api_key_secret, access_token=access_token, access_token_secret=access_token_secret) user_get = client.get_user(username='twitterusers') user_id = user_get.data.id print(user_id)  Above, is the code I've written so far in my attempt to create a Twitter bot that replies to a specific user's most recent tweet, however I did not make it far as I receive the following error when attempting to use the Client.get_user() method to retrieve a user's Twitter ID: tweepy.errors.Forbidden: 403 Forbidden When authenticating requests to the Twitter API v2 endpoints, you must use keys and tokens from a Twitter developer App that is attached to a Project. You can create a project via the developer portal. I went into the Twitter developer portal --> user authentication settings, and I made sure that the app permissions were set to 'Read and write'. I then tried regenerating all of my keys and tokens and made sure that there was no issue with the validity of the credentials themselves. I further confirmed it's not an issue with the keys or tokens when I found I was able to successfully tweet without receiving any errors using the following code. import tweepy import os api_key = os.getenv('TWI_API_KEY') api_key_secret = os.getenv('TWI_API_KEY_SECRET') bearer_token = os.getenv('TWI_BEARER_TOKEN') access_token = os.getenv('TWI_ACCESS_TOKEN') access_token_secret = os.getenv('TWI_ACCESS_TOKEN_SECRET') client = tweepy.Client(bearer_token=bearer_token, consumer_key=api_key, consumer_secret=api_key_secret, access_token=access_token, access_token_secret=access_token_secret) response = client.create_tweet(text='Testing!') print(response)  I tried changing the credentials passed to tweepy.Client(), first using the bearer_token only. Passing the bearer_token exclusively, as I have seen done in some examples, I received the same 403 Error as mentioned above. Then I tried passing the consumer_key, consumer_secret, access_token, access_token_secret. In this case, I receive the following 401 error: tweepy.errors.Unauthorized: 401 Unauthorized Unauthorized In either case, I am unable to utilize Client.get_user()

Answer: The error is because of twitter policy change where in the free tier, there is no way for us to pull tweets, we can only do that if we have a basic subscription which is 100$ per month now. Details can be found here: https://developer.twitter.com/en/products/twitter-api
Question: I have a program where the user selects a date from a datepicker, and I need to convert this date to another format. The original format is %d/%m/%Y and I need to convert it to %-d-%b-%Y. I made a small example of what happens: from datetime import datetime # Import tkinter library from tkinter import * from tkcalendar import Calendar, DateEntry win = Tk() win.geometry("750x250") win.title("Example") def convert(): date1 = cal.get() datetimeobject = datetime.strptime(date1, '%d/%m/%Y') print(date1) new_format = datetimeobject.strftime('%-d-%b-%Y') print(new_format) cal = DateEntry(win, width=16, background="gray61", foreground="white", bd=2, date_pattern='dd/mm/y') cal.pack(pady=20) btn = Button(win, command=convert, text='PRESS') btn.pack(pady=50) win.mainloop()  This gives me the following error:  File "---------\date.py", line 15, in convert new_format = datetimeobject.strftime('%-d-%b-%Y') ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ ValueError: Invalid format string 

Answer: The reason why that gives an error is because %-d only works on Unix machines (Linux, MacOS). On Windows (or Cygwin), you have to use %#d.
Question: What's the alternative of pandas : data['ColumnA'].str[:2]  in python polars? pl.col('ColumnA').str[:3]  throws TypeError: 'ExprStringNameSpace' object is not subscriptable  error.

Answer: You can use str.slice, It takes two arguments offset and length. offset specifies the start index and the length specifies the length of slice. If set to None (default), the slice is taken to the end of the string. >>> import polars as pl >>> >>> df = pl.DataFrame({"animal": ["Crab", "cat and dog", "rab$bit", None]}) >>> df shape: (4, 1) ┌─────────────┐ │ animal │ │ --- │ │ str │ ╞═════════════╡ │ Crab │ │ cat and dog │ │ rab$bit │ │ null │ └─────────────┘ >>> df.with_columns(pl.col("animal").str.slice(0, 3).alias("sub_string")) shape: (4, 2) ┌─────────────┬────────────┐ │ animal ┆ sub_string │ │ --- ┆ --- │ │ str ┆ str │ ╞═════════════╪════════════╡ │ Crab ┆ Cra │ │ cat and dog ┆ cat │ │ rab$bit ┆ rab │ │ null ┆ null │ └─────────────┴────────────┘
Question: Using pandas version 2, I get an error when calling iteritems. for event_id, region in column.iteritems(): pass  The following error message appears: Traceback (most recent call last): File "/home/analyst/anaconda3/envs/outrigger_env/lib/python3.10/site- packages/outrigger/io/gtf.py", line 185, in exon_bedfiles for event_id, region in column.iteritems()) AttributeError: 'Series' object has no attribute 'iteritems' 

Answer: iteritems was removed in 2.0.0 by GH45321.  Removed deprecated Series.iteritems()... use obj.items instead. (from v2.0.0 release notes)  You can use items wherever you used iteritems: for event_id, region in column.items(): pass
Question: I am new to transformers models and trying to extract aspect and sentiment for a sentence but having issues from transformers import AutoTokenizer, AutoModelForSequenceClassification model_name = "yangheng/deberta-v3-base-absa-v1.1" tokenizer = AutoTokenizer.from_pretrained(model_name) model = AutoModelForSequenceClassification.from_pretrained(model_name) text = "The food was great but the service was terrible." inputs = tokenizer(text, return_tensors="pt") outputs = model(**inputs)  I am able to get the tensor what I need is the output to extract the aspect and sentiment for the overall sentence I tried this however getting error sentiment_scores = outputs.logits.softmax(dim=1) aspect_scores = sentiment_scores[:, 1:-1] aspects = [tokenizer.decode([x]) for x in inputs["input_ids"].squeeze()][1:-1] sentiments = ['Positive' if score > 0.5 else 'Negative' for score in aspect_scores.squeeze()] for aspect, sentiment in zip(aspects, sentiments): print(f"{aspect}: {sentiment}")  I am looking for below o/p or similar o/p I am unable to write the logic as to how extract aspect and sentiment text -The food was great but the service was terrible aspect- food ,sentiment positive aspect - service, sentiment negative or at overall level aspect - food, sentiment positive 

Answer: The model you are trying to use predicts the sentiment for a given aspect based on a text. That means, it requires text and aspect to perform a prediction. It was not trained to extract aspects from a text. You could use a keyword extraction model to extract aspects (compare this SO answer). import torch import torch.nn.functional as F from transformers import AutoTokenizer, AutoModelForSequenceClassification model_name = "yangheng/deberta-v3-base-absa-v1.1" tokenizer = AutoTokenizer.from_pretrained(model_name) model = AutoModelForSequenceClassification.from_pretrained(model_name) aspects = ["food", "service"] text = "The food was great but the service was terrible." sentiment_aspect = {} for aspect in aspects: inputs = tokenizer(text, aspect, return_tensors="pt") with torch.inference_mode(): outputs = model(**inputs) scores = F.softmax(outputs.logits[0], dim=-1) label_id = torch.argmax(scores).item() sentiment_aspect[aspect] = (model.config.id2label[label_id], scores[label_id].item()) print(sentiment_aspect)  Output: {'food': ('Positive', 0.9973154664039612), 'service': ('Negative', 0.9935430288314819)}
Question: I have this output that was generated by model.predict() 0: 480x640 1 Hole, 234.1ms Speed: 3.0ms preprocess, 234.1ms inference, 4.0ms postprocess per image at shape (1, 3, 640, 640) 0: 480x640 1 Hole, 193.6ms Speed: 3.0ms preprocess, 193.6ms inference, 3.5ms postprocess per image at shape (1, 3, 640, 640) ...  How do I hide the output from the terminal? I can't find out the information in this official link: https://docs.ultralytics.com/modes/predict/#arguments

Answer: The Ultralytics docs are sadly not up to date, as of today. The right way of doing this is: from ultralytics import YOLO model = YOLO('yolov8m-seg.pt') results = model.predict(source='0', verbose=False) for result in results: masks = result.masks.masks print(masks.shape)  Notice the verbose=False argument. This won't print the default output ... 0: 480x640 1 Hole, 193.6ms Speed: 3.0ms preprocess, 193.6ms inference, 3.5ms postprocess per image at shape (1, 3, 640, 640) ...  In this case only: ... torch.Size([4, 480, 640]) ...
Question: Getting the error while trying to run a langchain code. ValueError: `run` not supported when there is not exactly one input key, got ['question', 'documents']. Traceback: File "c:\users\aviparna.biswas\appdata\local\programs\python\python37\lib\site-packages\streamlit\runtime\scriptrunner\script_runner.py", line 565, in _run_script exec(code, module.__dict__) File "D:\Python Projects\POC\Radium\Ana\app.py", line 49, in <module> answer = question_chain.run(formatted_prompt) File "c:\users\aviparna.biswas\appdata\local\programs\python\python37\lib\site-packages\langchain\chains\base.py", line 106, in run f"`run` not supported when there is not exactly one input key, got ['question', 'documents']."  My code is as follows. import os from apikey import apikey import streamlit as st from langchain.llms import OpenAI from langchain.prompts import PromptTemplate from langchain.chains import LLMChain, SequentialChain #from langchain.memory import ConversationBufferMemory from docx import Document os.environ['OPENAI_API_KEY'] = apikey # App framework st.title('🦜🔗 Colab Ana Answering Bot..') prompt = st.text_input('Plug in your question here') # Upload multiple documents uploaded_files = st.file_uploader("Choose your documents (docx files)", accept_multiple_files=True, type=['docx']) document_text = "" # Read and combine Word documents def read_docx(file): doc = Document(file) full_text = [] for paragraph in doc.paragraphs: full_text.append(paragraph.text) return '\n'.join(full_text) for file in uploaded_files: document_text += read_docx(file) + "\n\n" with st.expander('Contextual Prompt'): st.write(document_text) # Prompt template question_template = PromptTemplate( input_variables=['question', 'documents'], template='Given the following documents: {documents}. Answer the question: {question}' ) # Llms llm = OpenAI(temperature=0.9) question_chain = LLMChain(llm=llm, prompt=question_template, verbose=True, output_key='answer') # Show answer if there's a prompt and documents are uploaded if prompt and document_text: formatted_prompt = question_template.format(question=prompt, documents=document_text) answer = question_chain.run(formatted_prompt) st.write(answer['answer'])  I have gone through the documentations and even then I am getting the same error. I have already seen demos where multiple prompts are being taken by langchain.

Answer: For a prompt with multiple inputs, use predict() instead of run(), or just call the chain directly. (Note: Requires Python 3.8+) prompt_template = "Tell me a {adjective} joke and make it include a {profession}" llm_chain = LLMChain( llm=OpenAI(temperature=0.5), prompt=PromptTemplate.from_template(prompt_template) ) # Option 1 llm_chain(inputs={"adjective": "corny", "profession": "plumber"}) # Option 2 llm_chain.predict(adjective="corny", profession="plumber")  Also note that you only need to assign the PromptTemplate at the moment you're instantiating the LLMChain - after that you're just passing in the template variables - in your case, documents and question (instead of passing in the formatted template, as you have currently).