eshangj/stackoverflow_q_and_a_sample huggingface dataset:
Question: I was trying to solve this coding challenge: "Given an array of pairs, each pair (x, y), which are both integer, denotes coordinate of a point in Cartesian coordinate plane, find many groups of three points are collinear." It turns out this below is the correct algorithm: def gcd(x, y): if y == 0: return x else: return gcd(y, x % y) def how_many_3point_in_line( points ): ans = 0 n = len( points ) for i in range( n - 2 ): slopes = defaultdict( int ) for j in range( i + 1, n ): dy, dx = points[i][1] - points[j][1], points[i][0] - points[j][0] gcd_num = gcd( dx, dy ) slopes[( dy // gcd_num, dx // gcd_num )] += 1 for _, occ in slopes.items(): ans += math.comb( occ, 2 ) return ans  Apparently the gcd is used to represent slope, what issue does it address?

Answer: A clarification: Many thanks for fellow no comment's teaching, indeed the GCD in OP's posted solution is needed, and my original answer is wrong. Taking points = [(0, 0), (999999997, 999999998), (999999998, 999999999)] as an example, the difference between 999999997/999999998 and 999999998/999999999 appears only after so many decimal digits that floating point number does not include, leading to 999999997/999999998 == 999999998/999999999 being evaluated to be True. I also found this answer that nicely discussed about related topic. This answer has been marked as accepted so I cannot delete it; I'll keep it here as a counterexample.  Original answer ( wrong ): I guess the author of your code is trying to address inaccuracy of floating point representation. But that worry is actually unnecessary given that both x and y are integer, e.g., 3.0/9.0 == 9.0/27.0 will give you True, I recently studied on related topic and can hence confirm this. The only case where you (may or may not) get trouble is when either the denominator or numerator is floating point number, which does not apply in your question. So, you can modify the solution to simply use numeric slope, but you need to specially handle case where dx is 0, otherwise you get “divided by zero” exception. def how_many_3point_in_line( points ): ans = 0 n = len( points ) for i in range( n - 2 ): slopes = defaultdict( int ) numVertical = 0 for j in range( i + 1, n ): dy, dx = points[i][1] - points[j][1], points[i][0] - points[j][0] if dx != 0: slopes[dy / dx] += 1 else: numVertical += 1 for _, occ in slopes.items(): ans += math.comb( occ, 2 ) ans += math.comb( numVertical, 2 ) return ans
Question: Say we have a multi-dimensional list, but with random dimensions, like: [ [ [1, 2, [3, 4]], [[5, 6], 7] ], [8, 9, [10]] ]  Is there any short way to flatten everything and just get the list [1, 2, ..., 10] ? I know there's solution for list of lists, such as loops or comprehension list, but here we assume that we don't know the dimensions of the list, and that there can be different levels of nesting lists.

Answer: A recursive function can make quick work of this: test = [ [ [1, 2, [3, 4]], [[5, 6], 7] ], [8, 9, [10]] ] def flatten(l): for item in l: if isinstance(item, list): for i in flatten(item): yield i else: yield item print(list(flatten(test)))   [1, 2, 3, 4, 5, 6, 7, 8, 9, 10]  This function iterates through each item in the first layer of lists. If it finds something that isn't a list, it returns it with yield. If it finds another list, the function calls itself to iterate upon that list as well. What you end up with is each value inside every list returned back to the caller.
Question: Why are outputs in the two cases different. I am new to this library Case 1 import numpy as np np.random.seed(2) array = np.random.random((3,1)) print('Printing array : \n', array) print('printing array - 1 : \n',array-1)  Output : Printing array : [[0.4359949 ] [0.02592623] [0.54966248]] printing array - 1 : [[-0.5640051 ] [-0.97407377] [-0.45033752]]  This is ok as 1 is subtracted from each element Case 2 print('Printing array : \n', np.random.random ((3,1))-1) Output: Printing array : [[-0.56467761] [-0.5796322 ] [-0.66966518]]  Whay are the two outputs different? np.random.random ((3,1)) should be same in both cases ( same seed) and so subtracting 1 should produce the same output. what am I messing up? I ran the code as was expecting the same output in both cases

Answer: The reason why you got different arrays has been explained elaborately by @Jon Skeet.  One workaround is to customize a function by packing the random seed together with the random number generator function, e.g., def runif(shape, seed = 2): np.random.seed(seed) return np.random.random(shape) for iter in range(2): print(f'Print array{iter}: \n {runif((3,1))-iter} \n')  and you will see Print array0: [[0.4359949 ] [0.02592623] [0.54966248]] Print array1: [[-0.5640051 ] [-0.97407377] [-0.45033752]]
Question: I am working on this ml project and I need to install catboost and xgboost using pip. the xgboost got installed successfully but catboost keeps giving the same error: (venv) D:\ML bootcamp\mlproject>pip install catboost Collecting catboost Using cached catboost-1.2.7.tar.gz (71.5 MB) Installing build dependencies ... done Getting requirements to build wheel ... error error: subprocess-exited-with-error × Getting requirements to build wheel did not run successfully. │ exit code: 1 ╰─> [24 lines of output] Traceback (most recent call last): File "D:\ML bootcamp\mlproject\venv\Lib\site-packages\pip\_vendor\pyproject_hooks\_in_process\_in_process.py", line 389, in <module> main() ~~~~^^ File "D:\ML bootcamp\mlproject\venv\Lib\site-packages\pip\_vendor\pyproject_hooks\_in_process\_in_process.py", line 373, in main json_out["return_val"] = hook(**hook_input["kwargs"]) ~~~~^^^^^^^^^^^^^^^^^^^^^^^^ File "D:\ML bootcamp\mlproject\venv\Lib\site-packages\pip\_vendor\pyproject_hooks\_in_process\_in_process.py", line 143, in get_requires_for_build_wheel return hook(config_settings) File "C:\Users\Administrator\AppData\Local\Temp\pip-build-env-xzdiplgy\overlay\Lib\site-packages\setuptools\build_meta.py", line 334, in get_requires_for_build_wheel return self._get_build_requires(config_settings, requirements=[]) ~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "C:\Users\Administrator\AppData\Local\Temp\pip-build-env-xzdiplgy\overlay\Lib\site-packages\setuptools\build_meta.py", line 304, in _get_build_requires self.run_setup() ~~~~~~~~~~~~~~^^ File "C:\Users\Administrator\AppData\Local\Temp\pip-build-env-xzdiplgy\overlay\Lib\site-packages\setuptools\build_meta.py", line 522, in run_setup super().run_setup(setup_script=setup_script) ~~~~^^^^^^^^^^^^^^^^ File "<string>", line 733, in <module> File "<string>", line 205, in get_catboost_version KeyError: 'VERSION' [end of output] note: This error originates from a subprocess, and is likely not a problem with pip. error: subprocess-exited-with-error × Getting requirements to build wheel did not run successfully. │ exit code: 1 ╰─> See above for output. note: This error originates from a subprocess, and is likely not a problem with pip.  I have tried what I could, but I don't know what else to do to fix this. Please let me know if someone knows the solution to this.

Answer: According to the catboost installation docs  CatBoost Python package supports only CPython Python implementation with versions < 3.13. Version 3.13.x support is in progress.  Source: https://catboost.ai/docs/en/concepts/python-installation There is also an open issue for this on their github repo: https://github.com/catboost/catboost/issues/2748  A possible solution would be to use a version of python that catboost currently supports. e.g. 3.12.x
Question: As you see in the attached image, there was a Bearish Engulfing candle on December 18.  But when I see the same data and engulfing value, it shows something else, in fact 0. Below is the function computing, detecting engulfing candle pattern: def detect_engulfing_pattern(tsla_df): df = tsla_df.rename(columns={"Open": "open", "High": "high", "Low": "low", "Close": "close"}) df['engulfing'] = talib.CDLENGULFING(df['open'], df['high'], df['low'], df['close']) return df  

Answer: It’s not an engulfing pattern because it only compares one bar to the next, not across multiple bars. I highlighted the part that I think you missed:
Question: I have a requirement to read a file which is located inside a particular folder in a pod in AKS. My manual flow would be to:  exec into the pod with kubectl. cd to the directory where the file is located. cat the file to see it's contents.  I want to automate all this purely using python. I am able to do it with subprocess but that would work only on a machine which has azure and kubectl setup. Thus, I am looking for a purely pythonic way of doing this. I have looked into the Kubernetes client for Python but I am not able to find a way to do everything which I listed above.

Answer: To read a file which is located inside a particular folder in a pod in AKS via Python script, follow the below steps Assuming you have a valid aks cluster up and running, deploy a pod with your desired file. For example - apiVersion: v1 kind: Pod metadata: name: my-pod labels: app: my-app spec: containers: - name: my-container image: busybox command: ["/bin/sh", "-c", "echo 'Hello from AKS' > /data/file.txt && sleep 3600"] volumeMounts: - name: data-volume mountPath: "/data" volumes: - name: data-volume emptyDir: {}  kubectl apply -f pod.yaml kubectl get pods    This one says `Hello from AKS' and it should reflect the same when you read the file from the pod using python. Install / update the necessary dependencies pip install kubernetes Here's the script- from kubernetes import client, config, stream def read_file_from_pod(namespace: str, pod_name: str, container_name: str, file_path: str) -> str: try: config.load_incluster_config() except config.config_exception.ConfigException: config.load_kube_config() api_instance = client.CoreV1Api() command = ["cat", file_path] try: exec_response = stream.stream( api_instance.connect_get_namespaced_pod_exec, name=pod_name, namespace=namespace, command=command, container=container_name, stderr=True, stdin=False, stdout=True, tty=False, ) return exec_response except Exception as e: return f"Error reading file from pod: {str(e)}" if __name__ == "__main__": namespace = "default" pod_name = "my-pod" container_name = "my-container" file_path = "/data/file.txt" file_contents = read_file_from_pod(namespace, pod_name, container_name, file_path) print("File Contents:", file_contents)  Save and run the script. Now you can read a file from a pod in AKS in a Pythonic way.
Question: When trying to extract content from a MS Word .docx file using Azure Document Intelligence, I expected the returned response to contain a page element for each page in the document and for each of those page elements to contain multiple lines in line with the documentation. Instead, I always receive as a single page with no (None) lines and the entire document's contents as a list of words. Sample document:  Minimal reproducible example: from azure.core.credentials import AzureKeyCredential from azure.ai.documentintelligence import DocumentIntelligenceClient from azure.ai.documentintelligence.models import DocumentAnalysisFeature, AnalyzeResult, AnalyzeDocumentRequest def main(): client = DocumentIntelligenceClient( 'MY ENDPOINT', AzureKeyCredential('MY KEY') ) document = 'small_test_document.docx' with open(document, "rb") as f: poller = client.begin_analyze_document( "prebuilt-layout", analyze_request=f, content_type="application/octet-stream" ) result = poller.result() print(f'Found {len(result.pages)} page(s)') for page in result.pages: print(f'Page #{page.page_number}') print(f' {page.lines=}') print(f' {len(page.words)=}') if __name__ == '__main__': main()  Expected output: Found 2 page(s) Page #1 page.lines=6 len(page.words)=58 Page #2 page.lines=1 len(page.words)=8  Actual output: Found 1 page(s) Page #1 page.lines=None len(page.words)=66  My question is: Why, and what should I do differently to get the expected output?

Answer: As you have shown in your Actual output, all the 66 characters in your document are considered as one page. This is the expected behavior. As mentioned in the Docs on how the page units are computed: 3,000 characters are considered as one page unit in Word Document.    File format Computed page unit Total pages     Word (DOCX) Up to 3,000 characters = 1 page unit, embedded or linked images not supported Total pages of up to 3,000 characters each    So each 3000 characters is considered as 1 page. The page breaks in your document are not considered. Additionally the following features are not supported for the Microsoft Office (DOCX, XLSX, PPTX) and HTML files:  There are no angle, width/height and unit with each page object. For each object detected, there is no bounding polygon or bounding region. Page range (pages) is not supported as a parameter. No lines object.  Reference. So Document Intelligence has limited support for docx files. Your best option is to use the PDF files. You will get the content analyzed page by page in the PDF files. Not ideal, but if you do need to work with docx files, then first convert them to pdf files (using relevant API's) and process them with the Document Intelligence.
Question: I have this kind of array a = np.array([[-999, 9, 7, 3], [2, 1, -999, 1], [1, 5, 4, 6], [0, 6, -999, 9], [1, -999, -999, 6], [8, 4, 4, 8]])  I want to get 40% percentile of each row in that array where it is not equal -999 If I use np.percentile(a, 40, axis=1) I will get array([ 3.8, 1. , 4.2, 1.2, -799. , 4.8]) which is still include -999 the output I want will be like this [ 6.2, # 3 or 7 also ok 1, 4.2, # 4 or 5 also ok 4.8, # 0 or 6 also ok 1, 4 ]  Thank you

Answer: You can replace the -999s with NaNs and use nanpercentile. import numpy as np a = np.array([[-999, 9, 7, 3], [2, 1, -999, 1], [1, 5, 4, 6], [0, 6, -999, 9], [1, -999, -999, 6], [8, 4, 4, 8]], dtype=np.float64) a[a == -999] = np.nan np.nanpercentile(a, 40, axis=-1, keepdims=True) # array([[6.2], # [1. ], # [4.2], # [4.8], # [3. ], # [4.8]]) # Use the `method` argument if you want a different type of estimate # `keepdims=True` keeps the result a column, which it looks like you want  You asked for a solution "in NumPy", and that's it. (Unless you want to re-implement percentile, which is not so hard. Or I suppose you could use apply_along_axis on a function that removes the -999s before taking the quantile, but that will just loop in Python over the slices, which can be slow.)  If you don't want to have to change the dtype and replace with NaNs to perform the operation, you can use NumPy masked arrays with scipy.stats.mquantiles. import numpy as np from scipy import stats a = np.array([[-999, 9, 7, 3], [2, 1, -999, 1], [1, 5, 4, 6], [0, 6, -999, 9], [1, -999, -999, 6], [8, 4, 4, 8]]) mask = a == -999 b = np.ma.masked_array(a, mask=mask) stats.mstats.mquantiles(b, 0.4, alphap=1, betap=1, axis=-1) # alphap=1, betap=1 are the settings to reproduce the same values produced by NumPy's default `method`.  But beware that mquantiles is on its way out, superseded by new features in the next release.
Question: I want to drop the first group of rows based on a column's value. Here is an example of a table    stage h1 h2 h3     0 4 55 55   0 5 66 44   0 4 66 33   1 3 33 55   0 5 44 33    Get the column stage, get all the first group of rows that start with 0, and drop the rows in the table. The table will look like this:    stage h1 h2 h3     1 3 33 55   0 5 44 33    This is what I did: import pandas as pd data = {'stage': [0, 0, 0, 1, 0], 'h1': [4, 5, 4, 3, 5], 'h2': [55, 66, 66, 33, 44], 'h3': [55, 44, 33, 55, 33]} df = pd.DataFrame(data) # Find indices of the first group of rows with uiwp_washing_stage = 0 indices_to_drop = [] for i in range(len(df)): if df['stage'].iloc[i] == 0: indices_to_drop.append(i) else: break df = df.drop(indices_to_drop) df = df.reset_index(drop=True) print(df)  The above seems to work, but if the file is too big it takes a while, is there a Pands way of doing this?

Answer: df.iloc[df['stage'].diff().idxmax():]  First, find the first transition from 0 to 1 is by computing the difference between consecutive values in the stage column (using diff). Then use idxmax to locate the index where the first transition occurs. NB: In case, there are transitions that differ more than 1 unit, then use: df.iloc[df['stage'].diff().gt(0).idxmax():] Output:  stage h1 h2 h3 3 1 3 33 55 4 0 5 44 33
Question: I am using minmaxscaler trying to scaling each column. The scaled result for each column is always all zero. For example , below the values of df_test_1 after finishing scaling is all zero. But even with all values of zero, using inverse_transferm from this values of zero can still revert back to original values. But why the results of scaled are shown all zero? from sklearn.preprocessing import MinMaxScaler df_dict={'A':[-1,-0.5,0,1],'B':[2,6,10,18]} df_test=pd.DataFrame(df_dict) print('original scale data') print(df_test) scaler_model_list=[] df_test_1=df_test.copy() for col in df_test.columns: scaler = MinMaxScaler() scaler_model_list.append(scaler) # need to save scalerfor each column since there are different if we want to use inverse_transform() later df_test_1.loc[:,col]=scaler.fit_transform(df_test_1.loc[:,col].values.reshape(1,-1))[0] print('after finishing scaling') print(df_test_1) print('after inverse transformation') print(scaler_model_list[0].inverse_transform(df_test_1.iloc[:,0].values.reshape(1,-1))) print(scaler_model_list[1].inverse_transform(df_test_1.iloc[:,1].values.reshape(1,-1))) original scale data A B 0 -1.0 2 1 -0.5 6 2 0.0 10 3 1.0 18 after finishing scaling A B 0 0.0 0 1 0.0 0 2 0.0 0 3 0.0 0 after inverse transformation [[-1. -0.5 0. 1. ]] [[ 2. 6. 10. 18.]] 

Answer: According to MinMaxScaler DOC:  X : array-like of shape (n_samples, n_features) The data used to compute the per-feature minimum and maximum used for later scaling along the features axis.  When you reshape your data here: df_test_1.loc[:,df_test.columns[1]].values.reshape(1,-1)  you get 1 row data with 4 columns in your case (and only 1 value in each of them) instead of 1 column with 4 rows . You can fix your code one of the following ways:  Fix reshape dimensions: df_test_1.loc[:,col]=scaler.fit_transform(df_test_1.loc[:,col].values.reshape(-1,1)) Pass a dataframe instead of reshaped Series by indexing using list: df_test_1.loc[:,col]=scaler.fit_transform(df_test_1.loc[:,[col]]) Passing all required columns at once:  df_test_2=df_test.copy() scaler = MinMaxScaler() cols = df_test.columns df_test_2.loc[:,cols]=scaler.fit_transform(df_test_2.loc[:,cols])
Question: Say I have the following data: import duckdb rel = duckdb.sql(""" FROM VALUES ([{'a': 'foo', 'b': 'bta'}]), ([]), ([{'a': 'jun', 'b': 'jul'}, {'a':'nov', 'b': 'obt'}]) df(my_col) SELECT * """)  which looks like this: ┌──────────────────────────────────────────────┐ │ my_col │ │ struct(a varchar, b varchar)[] │ ├──────────────────────────────────────────────┤ │ [{'a': foo, 'b': bta}] │ │ [] │ │ [{'a': jun, 'b': jul}, {'a': nov, 'b': obt}] │ └──────────────────────────────────────────────┘  I would like to keep all rows where for any of the items in one of the elements of 'my_col', field 'a' contains the substring 'bt' So, expected output: ┌──────────────────────────────────────────────┐ │ my_col │ │ struct(a varchar, b varchar)[] │ ├──────────────────────────────────────────────┤ │ [{'a': foo, 'b': bta}] │ │ [{'a': jun, 'b': jul}, {'a': nov, 'b': obt}] │ └──────────────────────────────────────────────┘  How can I write a SQL query to do that?

Answer: Maybe list_sum() the bools or list_bool_or()?  https://duckdb.org/docs/stable/sql/functions/list.html#list_-rewrite-functions  duckdb.sql(""" FROM VALUES ([{'a': 'foo', 'b': 'bta'}]), ([]), ([{'a': 'jun', 'b': 'jul'}, {'a':'nov', 'b': 'obt'}]) df(my_col) SELECT * WHERE list_bool_or(['bt' in s.b for s in my_col]) """)  ┌──────────────────────────────────────────────┐ │ my_col │ │ struct(a varchar, b varchar)[] │ ├──────────────────────────────────────────────┤ │ [{'a': foo, 'b': bta}] │ │ [{'a': jun, 'b': jul}, {'a': nov, 'b': obt}] │ └──────────────────────────────────────────────┘  The list comprehension is the same as list_apply(my_col, s -> 'bt' in s.b)
Question: On the Pandas dataframe below I use max to find the maximum value: df["Max"] = df[['Day 1','Day 2','Day 4']].max(axis=1)     Car Day 1 Day 2 Day 4 Max     Car1 4 7 3 7   car2 8 2 1 8    What do I do to find when is the maximum value instead of the value itself as the table example below?    Car Day 1 Day 2 Day 4 When     Car1 4 7 3 Day 2   car2 8 2 1 Day 1   

Answer: Yes, you can achieve this in Pandas using .idxmax(axis=1), which returns the column name where the maximum value occurs. Documentation - pandas.DataFrame.idxmax import pandas as pd data = { 'Car': ['Car1', 'Car2'], 'Day 1': [4, 8], 'Day 2': [7, 2], 'Day 4': [3, 1] } df = pd.DataFrame(data) df.set_index('Car', inplace=True) # Find the column name where the maximum value occurs df['When'] = df.idxmax(axis=1) print(df)  Output:-  Day 1 Day 2 Day 4 When Car Car1 4 7 3 Day 2 Car2 8 2 1 Day 1
Question: Is it possible specify related model fields for selection with only() function in query? In this example I got KeyError: 'provider__description' from typing import Optional, List from tortoise import fields from models import BaseModel class Token(BaseModel): id = fields.CharField(primary_key=True, max_length=128) user = fields.ForeignKeyField( model_name="models.User", related_name="user_tokens", on_delete=fields.CASCADE, ) provider = fields.ForeignKeyField( model_name="models.Provider", related_name="provider_tokens", on_delete=fields.CASCADE, ) expires = fields.DatetimeField(default=0) class Provider(BaseModel): id = fields.IntField(pk=True) url = fields.CharField(max_length=255, unique=True, null=False) description = fields.CharField(max_length=255, null=False) inbound_id = fields.IntField(null=False) async def get_tokens(user_id: Optional[int] = None) -> list[Token]: query = Token.all().only("id", "provider__description", "expires").select_related("provider") if user_id: query = query.filter(user_id=user_id) return await query  With prefetch_related() got same error. For solve problem I tried select_relation() and prefetch_related() before only(). And tried

Answer: Found solve in tortoise-orm docs on Github. For prefetch only certain fields need to use tortoise.query_utils.Prefetch object. query = Token.all().prefetch_related( Prefetch("provider", queryset=Provider.all().only("id", "description")), )