eshangj/stackoverflow_q_and_a_sample huggingface dataset:
Question: I am experimenting with langchains and its applications, but as a newbie, I could not understand how the embeddings and indexing really work together here. I know what these two are, but I can't figure out a way to use the index that I created and saved using persist_directory. I succesfully saved the object created by VectorstoreIndexCreator using the following code: index = VectorstoreIndexCreator(vectorstore_kwargs={"persist_directory":"./custom_save_dir_path"}).from_loaders([loader])  but I cannot find a way to use the .pkl files created. How can I use these files in my chain to retrieve data? Also, how does the billing in openAI work? If I cannot use any saved embeddings or index, will it re-embed all the data every time I run the code? As a beginner, I am still learning my way around and any assistance would be greatly appreciated. Here is the full code: from langchain.document_loaders import CSVLoader from langchain.indexes import VectorstoreIndexCreator from langchain.chains import RetrievalQA from langchain.llms import OpenAI import os os.environ["OPENAI_API_KEY"] = "sk-xxx" # Load the documents loader = CSVLoader(file_path='data/data.csv') #creates an object with vectorstoreindexcreator index = VectorstoreIndexCreator(vectorstore_kwargs={"persist_directory":"./custom_save_dir_path"}).from_loaders([loader]) # Create a question-answering chain using the index chain = RetrievalQA.from_chain_type(llm=OpenAI(), chain_type="stuff", retriever=index.vectorstore.as_retriever(), input_key="question") # Pass a query to the chain while True: query = input("query: ") response = chain({"question": query}) print(response['result']) 

Answer: By default VectorstoreIndexCreator use the vector database DuckDB which is transient a keeps data in memory. If you want to persist data you have to use Chromadb and you need explicitly persist the data and load it when needed (for example load data when the db exists otherwise persist it). for more details about chromadb see: chroma The llm used in your case openai is the one responsible for the creation of embbedings (i.e the vectors that will be stored in the vector database). So whenever you process you data and store it in the vector store you will incure charges in openai, if you load vector store from the db you want incurr charge from openai
Question: I am trying to get some data from a url using the code below. I need help to produce the needed output.  from bs4 import BeautifulSoup import requests page = requests.get("https://coinmarketcap.com/new/") pagedata = page.text print ("* ", pagedata)  My output: (Truncated)  <a href="/currencies/cheems-token/" class="cmc-link"><div data-sensors-click="true" class="sc-aef7b723-0 sc-8497df48-0 iULUNk"> <img class="coin-logo" src="https://s2.coinmarketcap.com/static/img/coins/64x64/24988.png" alt="Cheems logo"/>< div class="sc-aef7b723-0 sc-8497df48-1 iATetF name-area "> <p font-weight="semibold" color="text" font-size="1" data-sensors-click="true" class="sc-4984dd93-0 kKpPOn">Cheems</p> <div data-nosnippet="true" class="sc-8497df48-2 gtUrBF"><div class="sc-8497df48-3 erCSsg">1</div> <p color="text3" class="sc-4984dd93-0 iqdbQL coin-item-symbol" font-size="1" data-sensors-click="true">Cheems</p></div></div></div></a></td> <td style="text-align:end"><span>$0.00000005918</span></td><td style="text-align:end"><span class="sc-97d6d2ca-0 bQjSqS"> <span class="icon-Caret-down"></span>19.90%</span></td><td style="text-align:end"><span class="sc-97d6d2ca-0 cYiHal"> <span class="icon-Caret-up"></span>58.53%</span></td><td style="text-align:end"><div class="sc-abe52752-3 eSDaBL">--</div></td> <td style="text-align:end">$3,774,982</td><td style="text-align:end"> <div class="sc-abe52752-3 eSDaBL"><img src="https://s2.coinmarketcap.com/static/img/coins/64x64/24091.png"/>zkSync</div></td> <td style="text-align:end">3 hours ago</td><td><div class="sc-aef7b723-0 dDQUel">  Output Wanted:  Name Symbol Price 1h 24h MarketCap Volume Blockchain Cheems Cheems $0.00000005918 19.90% 58.53% --- $3,774,982 zkSync CLIPS Clips $0.00005811 10.38% 9.44% $12,697,441 $45,436,152 Ethereum Pooh Inu POOH $0.0...07305 17.69% 87.04% $1,497,586 $332,306 BNB 

Answer: With pandas, you can use read_html with a bit of post-processing : #pip install pandas import pandas as pd usecols = ["Name", "Symbol", "Price", "1h", "24h", "MarketCap", "Volume", "Blockchain"] df = pd.read_html("https://coinmarketcap.com/new/")[0] df[["Name", "Symbol"]] = df["Name"].str.split(r"\d+", expand=True) df = df.rename(columns={"Fully Diluted Market Cap": "MarketCap"})[usecols]  Output : print(df) Name Symbol Price 1h 24h MarketCap Volume Blockchain 0 Cheems Cheems $0.00000006134 3.64% 64.31% -- $4,361,803 zkSync 1 Clips CLIPS $0.00005213 10.28% 18.75% $11,391,728 $43,346,358 Ethereum 2 Pooh Inu POOH $0.0...08057 10.29% 106.29% $1,651,664 $338,030 BNB .. ... ... ... ... ... ... ... ... 27 Mexican Pepe MEXPEPE $0.0000000617 19.14% 78.02% $61,704 $94,416 Ethereum 28 AIBabyDoge AIBABYDOGE $0.0...01419 7.50% 74.99% $595,903 $22,701 Ethereum 29 MonoLend MLD $0.143 1.38% 3.51% $285,937,514 $18,166 Polygon [30 rows x 8 columns]
Question: I have the following dataframe: import pandas as pd data = {'Test_Step_ID': ['9.1.1', '9.1.2', '9.1.3', '9.1.4'], 'Protocol_Name': ['A', 'B', 'C', 'D'], 'Req_ID': ['SRS_0081d', 'SRS_0079', 'SRS_0082SRS_0082a', 'SRS_0015SRS_0015cSRS_0015d'] } df = pd.DataFrame(data)  I want to duplicate the rows based on the column "Req_ID" based on the "SRS" value keeping all other columns values same; hence I want 2 rows for the SRS_0082, SRS_0082a and then three rows for SRS_0015, SRS_0015c, SRS_0015d Can someone help me here? appreciate the help. Thanks in advance. [EDITED]: I want the result to look like this: 

Answer: split on the zero width location between SRS and a preceding character using the '(?<=.)(?=SRS) regex, and explode: out = (df .assign(Req_ID=df['Req_ID'].str.split(r'(?<=.)(?=SRS)')) .explode('Req_ID') )  Output:  Test_Step_ID Protocol_Name Req_ID 0 9.1.1 A SRS_0081d 1 9.1.2 B SRS_0079 2 9.1.3 C SRS_0082 2 9.1.3 C SRS_0082a 3 9.1.4 D SRS_0015 3 9.1.4 D SRS_0015c 3 9.1.4 D SRS_0015d  Regex: (?<=.) # match any character before the split (?=SRS) # match "SRS" after the split  regex demo
Question: I have a list of values:  ['27', '27.1', '27.1.a', '27.1.b', '27.3.d.28', '27.3.d.28.1', '27.3.d.28.2', ]  I want to keep only the longest values for each "parent" value. That is, delete 27 if there is a 27.something. Delete 27.1. if there is a 27.1.something, etc. In this case, output should be ['27.1.a', '27.1.b', '27.3.d.28.1', '27.3.d.28.2', ]  Background info: this is a dataset in which 27 contains the sum of all 27.sth, so I only want to keep the most dissagregated data. I know that I can find the longest string by lis = ['27', '27.1', '27.1.a', '27.1.b', '27.3.d.28', '27.3.d.28.1', '27.3.d.28.2', ] ls = max(len(x) for x in lis) [x for x in lis if len(x) == le]  But I don't know how to do this tree search. I guess something with regex could work but I have no idea where to start. Any hint is appreciated. I also saw this idea which works well if I had only one level (one period), but I don't know how to apply it to my case.

Answer: Assuming the match is anchored on the left, a strategy might be to sort the strings, thus the strings will cluster by prefix, then only compare the successive pairs: def is_parent(p, target): return p.startswith(target) and len(p)>len(target) and p[len(target)] == '.' out = [] prev = '' for s in sorted(lis)+['']: if prev and not is_parent(s, prev): out.append(prev) prev = s print(out)  Output: ['27.1.a', '27.1.b', '27.3.d.28.1', '27.3.d.28.2']
Question: range() in Python seems to accept integer as parameter only. How to make range() accept a custom class as parameter? I've defined a class such as: class MyInteger: def __init__(self, a:int): self._a = a def __int__(self): return self._a  And I tried: n = MyInteger(5) for i in range(n): print(i)  It always causes 'object cannot be interpreted as an integer' whether or not I defined __int__. Is there any way to solve this problem? range(n) cannot be changed to range(int(n)) or range(n.get_int()).

Answer: Implement __index__ to allow your class instances to be interpreted as an int class MyInteger: def __init__(self, a:int): self._a = a def __index__(self): return self._a  n = MyInteger(5) for i in range(n): print(i)
Question: I have a Cloud Function in python 3.9 that calls this code : firebase_admin.initialize_app() def check_token(token, app_check_token): """ :param app_check_token: :param token: :return: """ try: app_token = app_check.verify_token(app_check_token) logging.info(f"App check token verified : {app_token}") except Exception as e: logging.error(f"Exception while decoding app check token : {e}") try: decoded_token = auth.verify_id_token(token) logging.info(f"verified token : {decoded_token}") if "uid" in decoded_token: return decoded_token["uid"] return "" except Exception as e: logging.error(f"check_token : {e}") return ""  And here is the log i have from Cloud logging when i call my function with a valid firebase id token : ERROR:root:check_token : 'HTTPResponse' object has no attribute 'strict'  What does it mean please ? Note : i have 10 cloud functions, only one has this issue and there is no differences with the others... ChatGPT says it's an error with Firebase authentication backend and to contact firbease support, but since this happens only in one of my Cloud Functions i would like to understand if i am doing something wrong. PS : If i run this cloud function locally everything is fine with  functions-framework --target function_name --debug --port=8080 with the exact same code.

Answer: I had a similar issue and someone pointed me to this: https://github.com/psf/requests/issues/6437 and also for firebase specifically: https://github.com/firebase/firebase-admin-python/issues/699 Apparently there's been a breaking change to the library and one solution is to use urllib3 2.0.0 as a workaround which I am yet to try (Not sure how given I'm dependent on Firebase). You can see my question here: Firebase Authentication: 'HTTPResponse' object has no attribute 'strict', status: error
Question: I have a Python string: test = 'blablabla_foo_bar_baz'  I want the negative index of the second occurence from the right of _: in this case, this would be -8, but even 13, the positive index, would be an acceptable answer, because it's not hard to compute the negative index from the positive index. How can I do that? A GPT-4 solution uses rfind + iteration, but I was wondering if it could be possible to solve it in one shot, without iteration. Of course the answer should handle strings of arbitrary length, and it should match an arbitrary character (_ in this example). EDIT: I need the negative index of the second occurence from the right of _, because I want to strip away _bar_baz, which can be done easily with test_prefix = test[:-8]  Thus, alternative solutions that allow me to get test_prefix without first finding the negative index of the second occurence from the right of _, would also be ok. Using the length of the substring _bar_baz (8 in this case) is not ok because I don't know the length of this substring. Instead computing the length of the substring from the second occurrence from the right of _, until the end of the string, would be ok. It doesn't seem any different from computing he negative index of the second occurence from the right of _, though. EDIT2: a commenter says that it's impossible to solve this without iteration. I'm not sure they're right, but I'll show a (GPT-4-generated) example of what I don't want: def find_second_last_occurrence(text, char='_'): last_occurrence = text.rfind(char) second_last_occurrence = text.rfind(char, 0, last_occurrence) return second_last_occurrence string = "blablabla_foo_bar_baz" result = find_second_last_occurrence(string, '_') print(result) # Output: 20  In this case, we had to call rfind twice. Is this really necessary? I don't think all possible solutions (including those based on re, rsplit, etc.) require calling a builtin twice. Note that I didn't include any code checking for the occurence of _ in the string, because I know for sure that my input string contains at least two occurences of _.

Answer: I am unsure the exact solution that is wanted but if the goal is just to split the string there is a much cleaner approach which can also give you the index it was cut at. test = 'blablabla_foo_bar_baz' result, *_ = test.rsplit('_', maxsplit=2) # split from the right two times print(result) # blablabla_foo print(len(result)) # 13 which is the index it was cut off at print(len(test) - len(result)) # 8 i.e. the negative index
Question: I'm trying to authenticate to twitter's new API (v2) using tweepy and retrieve tweets but encounter a strange error related to the authentication process. I'm currently using the free access to the API. Code sample : import tweepy # Authentification OAuth 1.0a User Context to retrieve my own data dict_twitter_api = { "consumer_key": "blah", "consumer_secret": "blah", "access_token": "blah", "access_token_secret": "blah" } client = tweepy.Client(**dict_twitter_api) # If you're working behind a corporate proxy, # client.session.proxies = { # "http": "my-corporate-proxy", # "https": "my-corporate-proxy", # } print(client.get_me()) # <-- this works well print(client.get_home_timeline())  Traceback result : > Forbidden: 403 Forbidden > When authenticating requests to the Twitter API v2 endpoints, you must use keys and tokens from a Twitter developer App that is attached to a Project. You can create a project via the developer portal.  I've checked my different tokens and confirmed that the OAuth 1.0a user context authentication should be working to retrieve my own timeline.

Answer: The returned exception is in fact not related to the authentication process. Twitter's free API access does not allow you to retrieve your own timeline or event tweets at all. This is stated in the about the twitter API documentation's page; it also explains why the page on rate limits only details the 'Basic' access'rates and bears no mention of the 'Free' access... Nota: in definitive, this appears to be a "no-code-related" question/answer; it may still prove usefull as 1) this is not referenced enough as yet and 2) the current traceback would throw any developer on the wrong path... Given the circonstances, I'll also share this resource which might be more uptodate/thorough.
Question: I need to debug a Python 3 program that uses the multiprocessing module. I want to keep track of which cores (of a multi-core machine) are getting used and how. Q: I am looking for a way for the Python code to determine which core is running it.  The closest I have found is to use the following: multiprocessing.current_process()._identity[0] - 1  Putting aside the fact that such code appears to be "going behind the API,"1 as far as I can tell, the code that initializes the _identity attribute makes no reference to the underlying hardware2, which I think is unsatisfactory.  1 For one thing, I can find no official documentation for the _identity attribute, as one would expect from the leading underscore in the name.2 More specifically, this code evaluates something like next(_process_counter), where _process_counter is initially set to the value itertools.count(1), and uses the the result as the basis for the _identity attribute's value.

Answer: This is absolutely going to be an OS-specific problem, but here's how to do it in Windows (source). As mentioned in the comments, the processor number will likely be changing all the time unless you specifically pin the task to a given core (outside the scope of this answer). from ctypes import * from ctypes import wintypes class PROCESSOR_NUMBER(Structure): _fields_ = [("Group", wintypes.WORD), ("Number", wintypes.BYTE), ("Reserved", wintypes.BYTE)] pn = PROCESSOR_NUMBER() windll.kernel32.GetCurrentProcessorNumberEx(byref(pn)) print(pn.Number)  Edit: for Ubuntu 20.04.5 (WSL) from ctypes import * libc = CDLL("libc.so.6") print(libc.sched_getcpu())
Question: I have some code similar to the one below. That code leaks, and I don't know why. The thing that leaks is a simple creation of a Python class' instance inside a C code. The function I use to check the leak is create_n_times that's defined below and just creates new Python instances and derefrences them in a loop. This is not an MWE per-se, but part of an example. To make it easier to understand, what the code does is:  The Python code defines the dataclass and registers it into the C-extension using set_ip_settings_type. Then, a C-extension function create_n_times is called and that function creates and destroys n instances of the Python dataclass.  Can anyone help? In Python: import c_api @dataclass class IpSettings: ip: str port: int dhcp: bool c_api.set_ip_settings_type(IpSettings) c_api.generate_n_times(100000)  In C++ I have the following code that's compiled into a Python extension called c_api (it's a part of that library's definition): #include <Python.h> // ... Other functions including a "PyInit" function extern "C" { PyObject* ip_settings_type = NULL; PyObject* set_ip_settings_type(PyObject* tp) { Py_XDECREF(ip_settings_type); Py_INCREF(tp); ip_settings_type = tp; return Py_None; } PyObject* create_n_times(PyObject* n) { long n_ = PyLong_AsLong(n); for (int i = 0; i < n_ ++i) { PyObject* factory_object = ip_settings_type; PyObject* args = PyTuple_New(3); PyTuple_SetItem(args, 0, PyUnicode_FromString("123.123.123.123")); PyTuple_SetItem(args, 1, PyLong_FromUnsignedLong(1231)); PyTuple_SetItem(args, 2, Py_False); PyObject* obj = PyObject_CallObject(factory_object, args); Py_DECREF(obj); } return Py_None; } } 

Answer: PyTuple_SetItem steals the reference to the supplied object, but Py_False is a single object. When the args tuple is destroyed, the reference count for Py_False isgetting mangled. Use PyBool_FromLong(0) to create a new reference to Py_False, like the other two calls to PyTuple_SetItem. (see docs.python.org/3/c-api/bool.html)
Question: I have an interger d and boolean array, say M = array([[ True, False, False, True], [False, True, False, False], [False, False, False, False], [False, False, False, True]])  Now I want to create a new array from M with the following rule: Trues stay and the next d positions left of a True also become True. I came up with this: newarray = np.full_like(M,False) for r,row in enumerate(M[1:],1): for i,boo in enumerate(row): if boo: newarray[max(0,r-d):r,i] = True  I have the feeling there is a more efficient way of doing this using numpy commands; probably using np.where. EDIT: e.g. d = 1, the result should be M = array([[ True, False, True, True], [ True, True, False, False], [False, False, False, False], [False, False, True, True]])  for d = 2, the result should be M = array([[ True, True, True, True], [ True, True, False, False], [False, False, False, False], [False, True, True, True]]) 

Answer: You can use a 2D convolution with scipy.signal.convolve2d: from scipy.signal import convolve2d d = 2 kernel = np.repeat([1, 1, 0], [d, 1, d])[None] # array([[1, 1, 1, 0, 0]]) out = convolve2d(M, kernel, mode='same') > 0  Output for d = 1: array([[ True, False, True, True], [ True, True, False, False], [False, False, False, False], [False, False, True, True]])  Output for d = 2: array([[ True, True, True, True], [ True, True, False, False], [False, False, False, False], [False, True, True, True]])
Question: I'm new to OPC UA and Python, but with the asyncua examples, I created the example that I need for a real project. Now I need to add security to the server and client, and for now, using a username and password is fine. Here is my functional code without security. If someone knows what functions I need to use to create two users, one with admin privileges and the other with none, please let me know. import asyncio from asyncua import Server, ua from asyncua.common.methods import uamethod from asyncua.common.structures104 import new_struct, new_struct_field @uamethod def func(parent, value): return value * 2 async def main(): server = Server() await server.init() server.set_endpoint("opc.tcp://localhost:4840/freeopcua/server/") uri = "http://examples.freeopcua.github.io" idx = await server.register_namespace(uri) myobj = await server.nodes.objects.add_object(idx, "MyObject") myvar = await myobj.add_variable(idx, "MyVariable", 0.0) await server.nodes.objects.add_method( ua.NodeId("ServerMethod", idx), ua.QualifiedName("ServerMethod", idx), func, [ua.VariantType.Int64], [ua.VariantType.Int64], ) struct = await new_struct(server, idx, "MyStruct", [ new_struct_field("FirstValue", ua.VariantType.Float, 0.0), new_struct_field("SecondValue", ua.VariantType.Float, 0.0), new_struct_field("ThirdValue", ua.VariantType.Float, 0.0), new_struct_field("FourthValue", ua.VariantType.Float, 0.0), new_struct_field("FifthValue", ua.VariantType.Float, 0.0), ]) custom_objs = await server.load_data_type_definitions() mystruct = await myobj.add_variable(idx, "my_struct", ua.Variant(ua.MyStruct(), ua.VariantType.ExtensionObject)) await mystruct.set_writable() await myvar.set_writable() print("Starting server!") async with server: while True: await asyncio.sleep(0.5) n_struct = await mystruct.get_value() var = await myvar.read_value() print ("\n%f\n%f\n%f\n%f\n%f\n%f" % (var, n_struct.FirstValue, n_struct.SecondValue, n_struct.ThirdValue, n_struct.FourthValue, n_struct.FifthValue)) try: loop = asyncio.get_running_loop() except RuntimeError: loop = None if loop and loop.is_running(): print('Async event loop already running. Adding coroutine to the event loop.') tsk = loop.create_task(main()) tsk.add_done_callback( lambda t: print(f'Task done with result={t.result()} << return val of main()')) else: print('Starting new event loop') result = asyncio.run(main(), debug=True)  I tried to use the encryption example from Asyncua, but I can't make it work. So, I tried to read the functions of the main code from Asyncua server and do something by myself, but I only got errors. Thank you.

Answer: You have to create a class that implements get_user class CustomUserManager: def get_user(self, iserver, username=None, password=None, certificate=None): if username == "admin": if password == 'secret_admin_pw': return User(role=UserRole.Admin) elif username == "user": if password == 'secret_pw': return User(role=UserRole.User) return None  To use the manager in your code: user_manager = CustomUserManager() server = Server(user_manager=cert_user_manager) await server.init() server.set_endpoint("opc.tcp://localhost:4840/freeopcua/server/")