eshangj/stackoverflow_q_and_a_sample huggingface dataset:
Question: i have dataframe with each row having a list value. id list_of_value 0 ['a','b','c'] 1 ['d','b','c'] 2 ['a','b','c'] 3 ['a','b','c']  i have to do a calculate a score with one row and against all the other rows For eg: Step 1: Take value of id 0: ['a','b','c'], Step 2: find the intersection between id 0 and id 1 , resultant = ['b','c'] Step 3: Score Calculation => resultant.size / id.size  repeat step 2,3 between id 0 and id 1,2,3, similarly for all the ids. and create a N x N dataframe; such as this: - 0 1 2 3 0 1 0.6 1 1 1 1 1 1 1 2 1 1 1 1 3 1 1 1 1  Right now my code has just one for loop: def scoreCalc(x,queryTData): #mathematical calculation commonTData = np.intersect1d(np.array(x),queryTData) return commonTData.size/queryTData.size ids = list(df['feed_id']) dfSim = pd.DataFrame() for indexQFID in range(len(ids)): queryTData = np.array(df.loc[df['id'] == ids[indexQFID]]['list_of_value'].values.tolist()) dfSim[segmentDfFeedIds[indexQFID]] = segmentDf['list_of_value'].apply(scoreCalc,args=(queryTData,))  Is there a better way to do this? can i just write one apply function instead doing a for-loop iteration. can i make it faster?

Answer: If you data is not too big, you can use get_dummies to encode the values and do a matrix multiplication: s = pd.get_dummies(df.list_of_value.explode()).sum(level=0) s.dot(s.T).div(s.sum(1))  Output:  0 1 2 3 0 1.000000 0.666667 1.000000 1.000000 1 0.666667 1.000000 0.666667 0.666667 2 1.000000 0.666667 1.000000 1.000000 3 1.000000 0.666667 1.000000 1.000000   Update: Here's a short explanation for the code. The main idea is to turn the given lists into one-hot-encoded:  a b c d 0 1 1 1 0 1 0 1 1 1 2 1 1 1 0 3 1 1 1 0  Once we have that, the size of intersection of the two rows, say, 0 and 1 is just their dot product, because a character belongs to both rows if and only if it is represented by 1 in both. With that in mind, first use df.list_of_value.explode()  to turn each cell into a series and concatenate all of those series. Output: 0 a 0 b 0 c 1 d 1 b 1 c 2 a 2 b 2 c 3 a 3 b 3 c Name: list_of_value, dtype: object  Now, we use pd.get_dummies on that series to turn it to a one-hot-encoded dataframe:  a b c d 0 1 0 0 0 0 0 1 0 0 0 0 0 1 0 1 0 0 0 1 1 0 1 0 0 1 0 0 1 0 2 1 0 0 0 2 0 1 0 0 2 0 0 1 0 3 1 0 0 0 3 0 1 0 0 3 0 0 1 0  As you can see, each value has its own row. Since we want to combine those belong to the same original row to one row, we can just sum them by the original index. Thus s = pd.get_dummies(df.list_of_value.explode()).sum(level=0)  gives the binary-encoded dataframe we want. The next line s.dot(s.T).div(s.sum(1))  is just as your logic: s.dot(s.T) computes dot products by rows, then .div(s.sum(1)) divides counts by rows.
Question: When I try to run this code as is I get this error "IDispatch = pythoncom.CoCreateInstance(IDispatch, None, clsctx, pythoncom.IID_IDispatch) pywintypes.com_error: (-2147221008, 'CoInitialize has not been called.', None, None)" , however if I run stp_tracker alone it works fine and if I run notify stp's alone it works just fine. I appreciate anyones input. Thanks import time import win32com.client # import sys from watchdog.observers import Observer from watchdog.events import PatternMatchingEventHandler # import watchdog class MyHandler(PatternMatchingEventHandler): patterns = ["*.stp", "*.step", "*.txt"] def process(self, event): """ event.event_type 'modified' | 'created' | 'moved' | 'deleted' event.is_directory True | False event.src_path path/to/observed/file """ # the file will be processed there print(event.src_path, event.event_type) def on_modified(self, event): self.process(event) notify_stps() def on_created(self, event): self.process(event) notify_stps() def on_deleted(self, event): self.process(event) notify_stps() def stp_tracker(): # /if __name__ == '__main__': path = r"W:\TestFolder" observer = Observer() observer.schedule(MyHandler(), path) observer.start() try: while True: time.sleep(1) except KeyboardInterrupt: observer.stop() observer.join() def notify_stps(): const = win32com.client.constants olMailItem = 0x0 obj = win32com.client.Dispatch("Outlook.Application") newMail = obj.CreateItem(olMailItem) newMail.Subject = "I AM SUBJECT!!" newMail.Body = "Step files in directory" # newMail.BodyFormat = 2 # olFormatHTML https://msdn.microsoft.com/en-us/library/office/aa219371(v=office.11).aspx # newMail.HTMLBody = "<HTML><BODY>Enter the <span style='color:red'>message</span> text here.</BODY></HTML>" newMail.To = 'Acoker251@outlook.com' # attachment1 = r"C:\Temp\example.pdf" # newMail.Attachments.Add(Source=attachment1) newMail.Send() stp_tracker() 

Answer: Apologize for that, but searching the internet and I found something that helped. I came across the same post earlier and assumed it was deprecated info because my AutoComplete in pycharm was not picking anything up when typing pythoncom.CoInitialize() so it made me think it was outdated info. Also the same information Strive Sun explained
Question: Is there a built-in way in SQLite (or similar) to keep the best of both worlds SQL / NoSQL, for small projects, i.e.:  stored in a (flat) file like SQLite (no client/server scheme, no server to install; more precisely : nothing else to install except pip install <package>) possibility to store rows as dict, without having a common structure for each row, like NoSQL databases support of simple queries  Example: db = NoSQLite('test.db') db.addrow({'name': 'john doe', 'balance': 1000, 'data': [1, 73.23, 18]}) db.addrow({'name': 'alice', 'balance': 2000, 'email': 'a@b.com'}) for row in db.find('balance > 1500'): print(row) # {'id': 'f565a9fd3a', 'name': 'alice', 'balance': 2000, 'email': 'a@b.com'} # id was auto-generated  Note: I have constantly been amazed along the years by how many interesting features are in fact possible with SQLite in a few lines of code, that's why I'm asking if what I describe here could maybe be available simply with SQLite by using only a few SQLite core features. PS: shelve could look like a solution but in fact it's just a persistent key/value store, and it doesn't have query/find functions; also bsddb (BerkeleyDB for Python) looks deprecated and has no query feature with a similar API.

Answer: It's possible via using the JSON1 extension to query JSON data stored in a column, yes: sqlite> CREATE TABLE test(data TEXT); sqlite> INSERT INTO test VALUES ('{"name":"john doe","balance":1000,"data":[1,73.23,18]}'); sqlite> INSERT INTO test VALUES ('{"name":"alice","balance":2000,"email":"a@b.com"}'); sqlite> SELECT * FROM test WHERE json_extract(data, '$.balance') > 1500; data -------------------------------------------------- {"name":"alice","balance":2000,"email":"a@b.com"}   If you're going to be querying the same field a lot, you can make it more efficient by adding an index on the expression: CREATE INDEX test_idx_balance ON test(json_extract(data, '$.balance'));  will use that index on the above query instead of scanning every single row.
Question: I want to get the indices of an xarray data array where some condition is satisfied. An answer provided in a related thread (here) for how to find the location for the maximum did not work for me either. In my case, I want to find out the locations for other types of conditions too, not just maximum. Here is what I tried: h=xr.DataArray(np.random.randn(3,4)) h.where(h==h.max(),drop=True).squeeze() # This is the output I got: <xarray.DataArray ()> array(1.66065694)  This does not return the position as shown in the example I linked to, even though I am executing the same command. Am not sure why.

Answer: I updated the linked example to show the indexes more clearly. Because xarray no longer adds default indexes, the previous example finds the max location but doesn't show the indexes. Copied below: In [17]: da = xr.DataArray( np.random.rand(2,3), dims=list('ab'), coords=dict(a=list('xy'), b=list('ijk')) ) In [18]: da.where(da==da.max(), drop=True).squeeze() Out[18]: <xarray.DataArray ()> array(0.96213673) Coordinates: a <U1 'x' b <U1 'j'
Question: I am probably looking right over it in the documentation, but I wanted to know if there is a way with XGBoost to generate both the prediction and probability for the results? In my case, I am trying to predict a multi-class classifier. it would be great if I could return Medium - 88%.   Classifier = Medium  Probability of Prediction = 88%  parameters params = { 'max_depth': 3, 'objective': 'multi:softmax', # error evaluation for multiclass training 'num_class': 3, 'n_gpus': 0 }  prediction pred = model.predict(D_test)  results array([2., 2., 1., ..., 1., 2., 2.], dtype=float32)  User friendly (label encoder) pred_int = pred.astype(int) label_encoder.inverse_transform(pred_int[:5]) array(['Medium', 'Medium', 'Low', 'Low', 'Medium'], dtype=object)  EDIT: @Reveille suggested predict_proba. I am not instantiating XGBClassifer(). Should I be? How would I modify my pipeline to use that, if so? params = { 'max_depth': 3, 'objective': 'multi:softmax', # error evaluation for multiclass training 'num_class': 3, 'n_gpus': 0 } steps = 20 # The number of training iterations model = xgb.train(params, D_train, steps) 

Answer: You can try pred_p = model.predict_proba(D_test)  An example I had around (not multi-class though): import xgboost as xgb from sklearn.datasets import make_moons from sklearn.model_selection import train_test_split X, y = make_moons(noise=0.3, random_state=0) X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.1) xgb_clf = xgb.XGBClassifier() xgb_clf = xgb_clf.fit(X_train, y_train) print(xgb_clf.predict(X_test)) print(xgb_clf.predict_proba(X_test)) [1 1 1 0 1 0 1 0 0 1] [[0.0394336 0.9605664 ] [0.03201818 0.9679818 ] [0.1275925 0.8724075 ] [0.94218 0.05782 ] [0.01464975 0.98535025] [0.966953 0.03304701] [0.01640552 0.9835945 ] [0.9297296 0.07027044] [0.9580196 0.0419804 ] [0.02849442 0.9715056 ]]  Note as mentioned in the comments by @scarpacci (ref):   predict_proba() method only exists for the scikit-learn interface
Question: I need a DateInput field in a ModelForm with the default HTML datepicker (I'm not using 3rd party libraries).  Since the DateInput is rendered with <input type = "text"> by default, the datepicker is missing (it comes for free with <input type = "date">)  I've found some examples explaining how to change the input type by handling widget parameters (below the code I've done so far) The issue I have the datepicker working correctly but in "update mode" when passing initial date value to the form (see view part), the date remains empty in the HTML.  I've tried to find the cause and it seems that the 'type': 'date' part in the widget customization is clearing the initial value is some way; in fact, removing it, the initial value date is displayed again, but I loose the datepicker of course. In the view the date is passed with a valid value I also found another similar unanswered question where the field was declared as class DateInput(forms.DateInput): input_type = 'date' date_effet = forms.DateField(widget=forms.DateInput(format='%d-%m-%Y'), label='Date effet')  the problem still remains My code model.py class TimesheetItem(models.Model): date = models.DateField() description = models.CharField(max_length=100) # many other fields here  form.py class TimesheetItemForm(forms.ModelForm): def __init__(self, *args, **kwargs): super().__init__(*args, **kwargs) # init is used for other fields initialization and crispy forms class Meta: model = TimesheetItem fields = ['date', 'description'] widgets = { 'date': forms.DateInput( format=('%d/%m/%Y'), attrs={'class': 'form-control', 'placeholder': 'Select a date', 'type': 'date' # <--- IF I REMOVE THIS LINE, THE INITIAL VALUE IS DISPLAYED }), }   view.py def edit(request, uuid): try: timesheet_entry_item = TimesheetItem.objects.get(uuid=uuid) if request.method == 'POST': form = TimesheetItemForm( data=request.POST, instance=timesheet_entry_item ) if form.is_valid(): pass # save the form else: form = TimesheetItemForm(initial={ 'date': timesheet_entry_item.date, # <--- the date here has a valid value 'description': timesheet_entry_item.description }) return render(request, 'template.html', {'form': form}) except ObjectDoesNotExist: raise Http404("error")  Thanks for any help M.

Answer: I managed to make it work. Following the cause of the issue, I hope it can be useful to others. The HTML <input type='date'> element wants a date in the format YYYY-mm-dd; in fact an example of working HTML must be like this: <input type="date" name="date" value="2020-03-31" class="form-control dateinput form-control" placeholder="Select a date" required="" id="id_date">  Since by default the form.DateInput produces the element <input type='text'>, it expects a date in the local format: let's say '31/03/2020'. Forcing the 'type': 'date' and local format format=('%d/%m/%Y') or not passing a format at all, it ignores the value passed since the <input type='date'> wants format=('%Y-%m-%d') At last the correct instruction was: widgets = { 'date': forms.DateInput( format=('%Y-%m-%d'), attrs={'class': 'form-control', 'placeholder': 'Select a date', 'type': 'date' }), }
Question: Let's say I have the following structure:  dir_1 ├── functions.py └── dir_2 └── code.ipynb  In, code.ipynb, I simply want to access a function inside functions.py and tried this: from ..functions import some_function  I get the error: attempted relative import with no known parent package  I have checked a bunch of similar posts but not yet figured this out... I am running jupyter notebook from a conda env and my python version is 3.7.6.

Answer: In your notebook do: import os, sys dir2 = os.path.abspath('') dir1 = os.path.dirname(dir2) if not dir1 in sys.path: sys.path.append(dir1) from functions import some_function
Question:  I've just started using Pyright. Running it on files that run perfectly well I get plenty of errors. This question is similar, but refers to one's own modules. For example Import "numpy" could not be resolved. What does it mean, and how do I resolve it?

Answer: On my computer I have 3 Pythons, a 3.6 from Anaconda, and a 2.7 & 3.7 that are regular python. Prompted by a nudge from this GH issue, I switched from the Anaconda 3.6 to the 3.7, and back again, and the problem went away.   I think that this is the case because your .vscode/settings.json (the following is mine), doesn't have that last line until you change your python, at which point, that last line is put in and Pyright has something to look at. { "python.linting.enabled": true, "python.formatting.provider": "black", "python.pythonPath": "C:\\Users\\ben\\Anaconda3\\python.exe" }
Question: I am using django 3.0 and I am trying to display a datepicker widget in my ModelForm, but I can't figure out how (all I can get is text field). I have tried looking for some solutions, but couldn't find any. This is how my Model and my ModelForm look like: class Membership(models.Model): start_date = models.DateField(default=datetime.today, null=True) owner = models.ForeignKey(Client, on_delete=models.CASCADE, null=True) type = models.ForeignKey(MembershipType, on_delete=models.CASCADE, null=True) class MembershipForm(ModelForm): class Meta: model = Membership fields = ['owner', 'start_date', 'type'] widgets = { 'start_date': forms.DateInput }  And this is my html: <form class="container" action="" method="POST"> {% csrf_token %} {{ form|crispy }} <button type="submit" class="btn btn-primary">Submit</button> </form> 

Answer: This is the expected behavior. A DateInput widget [Django-doc] is just a <input type="text"> element with an optional format parameter. You can make use of a package, like for example django-bootstrap-datepicker-plus [pypi] , and then define a form with the DatePickerInput: from bootstrap_datepicker_plus import DatePickerInput class MembershipForm(ModelForm): class Meta: model = Membership fields = ['owner', 'start_date', 'type'] widgets = { 'start_date': DatePickerInput } In the template you will need to render the media of the form and load the bootstrap css and javascript: {% load bootstrap4 %} {% bootstrap_css %} {% bootstrap_javascript jquery='full' %} {{ form.media }} <form class="container" action="" method="POST"> {% csrf_token %} {{ form|crispy }} <button type="submit" class="btn btn-primary">Submit</button> </form>
Question: Am attempting a tqdm progress bar with asyncio tasks gathered.  Want the progress bar to be progressively updated upon completion of a task. Tried the code: import asyncio import tqdm import random async def factorial(name, number): f = 1 for i in range(2, number+1): await asyncio.sleep(random.random()) f *= i print(f"Task {name}: factorial {number} = {f}") async def tq(flen): for _ in tqdm.tqdm(range(flen)): await asyncio.sleep(0.1) async def main(): # Schedule the three concurrently flist = [factorial("A", 2), factorial("B", 3), factorial("C", 4)] await asyncio.gather(*flist, tq(len(flist))) asyncio.run(main())  ...but this simply completes the tqdm bar and then processes factorials. Is there a way to make the progress bar move upon completion of each asyncio task?

Answer: Made a couple of small changes to Dragos' code in pbar format and used tqdm.write() to get almost what I want, as follows: import asyncio import random import tqdm async def factorial(name, number): f = 1 for i in range(2, number + 1): await asyncio.sleep(random.random()) f *= i return f"Task {name}: factorial {number} = {f}" async def tq(flen): for _ in tqdm.tqdm(range(flen)): await asyncio.sleep(0.1) async def main(): flist = [factorial("A", 2), factorial("B", 3), factorial("C", 4)] pbar = tqdm.tqdm(total=len(flist), position=0, ncols=90) for f in asyncio.as_completed(flist): value = await f pbar.set_description(desc=value, refresh=True) tqdm.tqdm.write(value) pbar.update() if __name__ == '__main__': asyncio.run(main())
Question: I would like to use pytest monkeypatch to mock a class which is imported into a separate module. Is this actually possible, and if so how does one do it? It seems like I have not seen an example for this exact situation. Suppose you have app with and imported class A in something.py from something import A #Class is imported class B : def __init__(self) : self.instance = A() #class instance is created def f(self, value) : return self.instance.g(value)  inside my test.py I want to mock A inside B from something import B #this is where I would mock A such that def mock_A : def g(self, value) : return 2*value #Then I would call B c = B() print(c.g(2)) #would be 4  I see how monkeypatch can be used to patch instances of classes, but how is it done for classes that have not yet been instantiated? Is it possible? Thanks!

Answer: tested this, works for me: def test_thing(monkeypatch): def patched_g(self, value): return value * 2 monkeypatch.setattr(A, 'g', patched_g) b = B() assert b.f(2) == 4
Question: I am using python-social-auth in my django project to use social platforms for authentication in my project. It all worked well but am getting this error ModuleNotFoundError: No module named 'jose' This is the whole error: [05/Apr/2020 14:01:00] "GET /accounts/login/ HTTP/1.1" 200 3058 Internal Server Error: /login/twitter/ Traceback (most recent call last): File "C:\Program Files\Python37\lib\site-packages\social_core\backends\utils.py", line 50, in get_backend return BACKENDSCACHE[name] KeyError: 'twitter' During handling of the above exception, another exception occurred: Traceback (most recent call last): File "C:\Users\Ahmed\AppData\Roaming\Python\Python37\site-packages\django\core\handlers\exception.py", line 34, in inner response = get_response(request) File "C:\Users\Ahmed\AppData\Roaming\Python\Python37\site-packages\django\core\handlers\base.py", line 115, in _get_response response = self.process_exception_by_middleware(e, request) File "C:\Users\Ahmed\AppData\Roaming\Python\Python37\site-packages\django\core\handlers\base.py", line 113, in _get_response response = wrapped_callback(request, *callback_args, **callback_kwargs) File "C:\Users\Ahmed\AppData\Roaming\Python\Python37\site-packages\django\views\decorators\cache.py", line 44, in _wrapped_view_func response = view_func(request, *args, **kwargs) File "C:\Program Files\Python37\lib\site-packages\social_django\utils.py", line 46, in wrapper backend, uri) File "C:\Program Files\Python37\lib\site-packages\social_django\utils.py", line 27, in load_backend Backend = get_backend(BACKENDS, name) File "C:\Program Files\Python37\lib\site-packages\social_core\backends\utils.py", line 53, in get_backend load_backends(backends, force_load=True) File "C:\Program Files\Python37\lib\site-packages\social_core\backends\utils.py", line 35, in load_backends backend = module_member(auth_backend) File "C:\Program Files\Python37\lib\site-packages\social_core\utils.py", line 62, in module_member module = import_module(mod) File "C:\Program Files\Python37\lib\site-packages\social_core\utils.py", line 56, in import_module __import__(name) File "C:\Program Files\Python37\lib\site-packages\social\backends\google.py", line 3, in <module> from social_core.backends.google_openidconnect import GoogleOpenIdConnect File "C:\Program Files\Python37\lib\site-packages\social_core\backends\google_openidconnect.py", line 5, in <module> from .open_id_connect import OpenIdConnectAuth File "C:\Program Files\Python37\lib\site-packages\social_core\backends\open_id_connect.py", line 6, in <module> from jose import jwk, jwt ModuleNotFoundError: No module named 'jose' [05/Apr/2020 14:01:06] "GET /login/twitter/?next=/accounts/login/ HTTP/1.1" 500 132103  I am pretty new to python and can't figure out what the issue is.

Answer: Install jose by running: pip install python-jose>=3.0.0
Question: I have some data, more or less like this: [ {"tag": "A", "level":0}, {"tag": "B", "level":1}, {"tag": "D", "level":2}, {"tag": "F", "level":3}, {"tag": "G", "level":4}, {"tag": "E", "level":2}, {"tag": "H", "level":3}, {"tag": "I", "level":3}, {"tag": "C", "level":1}, {"tag": "J", "level":2}, ]  I want to turn it into a multilevel dict based on depth level (key "level"): { "A": {"level": 0, "children": { "B": {"level": 1, "children": { "D": {"level": 2, "children": { "F": {"level": 3, "children": { "G": {"level": 4, "children": {}}}}}}, "E": {"level": 2, "children": { "H": {"level": 3, "children": {}}, "I": {"level": 3, "children": {}}}}}}, "C": {"level": 1, "children": { "J": {"level": 2, "children": {}}}}}} }  All I can come up with right now is this little piece of code... which obviously breaks after few items: def list2multilevel(list): children = {} parent = list.pop(0) tag = parent.get("Tag") level = parent.get("Level") for child in list: ctag = child.get("Tag") clevel = child.get("Level") if clevel == level + 1: children.update(list2multilevel(list)) elif clevel <= level: print(clevel, level) break return {tag: children}  Originally sat down to it on Friday and it was supposed to be just a small exercise....

Answer: data = [ {"tag": "A", "level": 0}, {"tag": "B", "level": 1}, {"tag": "D", "level": 2}, {"tag": "F", "level": 3}, {"tag": "G", "level": 4}, {"tag": "E", "level": 2}, {"tag": "H", "level": 3}, {"tag": "I", "level": 3}, {"tag": "C", "level": 1}, {"tag": "J", "level": 2}, ] root = {'level': -1, 'children': {}} parents = {-1: root} for datum in data: level = datum['level'] parents[level] = parents[level - 1]['children'][datum['tag']] = { 'level': datum['level'], 'children': {}, } result = root['children'] print(result)  output: {'A': {'level': 0, 'children': {'B': {'level': 1, 'children': {'D': {'level': 2, 'children': {'F': {'level': 3, 'children': {'G': {'level': 4, 'children': {}}}}}}, 'E': {'level': 2, 'children': {'H': {'level': 3, 'children': {}}, 'I': {'level': 3, 'children': {}}}}}}, 'C': {'level': 1, 'children': {'J': {'level': 2, 'children': {}}}}}}}  restriction:  level >= 0 Any level cannot be bigger than +1 of max level appeared before.  explanation:  parents is a dictionary to remember last element for each level. root is a starting point(dummy element). logic:  Start with -1 level which indicates the root. Make an item and register it into parent's children. Update same item to parents dictionary. Repeat. Extract root['children'].