ML Math


# ML Math
1. [Linear Algebra Data Structures](#linear-algebra-data-structures)
1. [Tensor Operations](#tensor-operations)
1. [Matrix Properties](#matrix-properties)
1. [Eigenvectors and Eigenvalues](#eigenvectors-and-eigenvalues)
1. [Matrix Operations for Machine Learning](#matrix-operations-for-machine-learning)
1. [Limits](#limits)
1. [Derivative and Differentiation](#derivative-and-differentiation)
1. [Automatic Differentiation](#automatic-differentiation)
1. [Partial-Derivative Calculus](#partial-derivative-calculus)
1. [Integral Calculus](#integral-calculus)
1. [Reference](#reference)


<a id="#Linear-Algebra-Data-Structures]"></a>
# Linear Algebra Data Structures

|Dimensions|Mathematical Name|Description|
|:-:|:-:|:-:|
|0 | scalar | magnitude only |
| 1 | vector | array |
| 2 | matrix | flat table, e.g., square |
| 3 | 3-tensor | 3D tables, e.g., cube |
| n | n-tensor | higher dimensional|


|Scalars|Vectors|Matrices|Tensors|
|:-:|:-:|:-:|:-:|
| No-D | 1D | 2D | 3D |
| x | ***x*** | ***X*** | .|
|![image.png](https://qiita-image-store.s3.ap-northeast-1.amazonaws.com/0/1089224/c4f856ab-4dda-3582-3b29-ec6a89fb5ca1.png)|![image.png](https://qiita-image-store.s3.ap-northeast-1.amazonaws.com/0/1089224/70fe781a-1c15-7d6b-d691-f66934c7a08a.png)|![image.png](https://qiita-image-store.s3.ap-northeast-1.amazonaws.com/0/1089224/f43f9e85-244a-3ddb-63a0-4444029c125e.png)| .|
| ![image.png](https://qiita-image-store.s3.ap-northeast-1.amazonaws.com/0/1089224/5efd2eb1-0ff9-b6c7-ed3c-482bcd5fefe3.png)| ![image.png](https://qiita-image-store.s3.ap-northeast-1.amazonaws.com/0/1089224/b3bd10d9-9121-314f-f148-b6b9f7f07fe8.png)|![image.png](https://qiita-image-store.s3.ap-northeast-1.amazonaws.com/0/1089224/239c2f7d-5235-f52a-32f8-5aede2c9cd7a.png)|![image.png](https://qiita-image-store.s3.ap-northeast-1.amazonaws.com/0/1089224/a2a8a928-f485-ef2b-e0a2-0e8cb93d90c2.png)|
| -  | Vector of Length n represents location in n dimensional space |(n_row, n_col) e.g., ```x[0:2], 0:2]```| .|

### NORM
- *Normalization helps you to scale down your features between 0 to 1*
- Norms are used to express distances, e.g., this vector and this vector are this far apart, according to this norm.
- If you use l2-normalization, “***UNIT NORM***” means that if we squared each element in the vector, and summed them, it would equal ***1***. 
  
|L^2|L^1|Squared L^2|Max Norm (L∞)| Generalized L^p Norm |
|:-:|:-:|:-:|:-:|:-:|
| ![image.png](https://qiita-image-store.s3.ap-northeast-1.amazonaws.com/0/1089224/abc7d0e6-02fe-884c-27c3-8dab8912d8c1.png)| ![image.png](https://qiita-image-store.s3.ap-northeast-1.amazonaws.com/0/1089224/ee9a916d-8fe3-0e4c-efd8-1f3888d5b300.png) | ![image.png](https://qiita-image-store.s3.ap-northeast-1.amazonaws.com/0/1089224/9b062883-2757-2296-e7ed-e38b1f43727c.png) | ![image.png](https://qiita-image-store.s3.ap-northeast-1.amazonaws.com/0/1089224/d5719e9e-9bb0-4d4f-2ede-cbfdfb83b035.png) | ![image.png](https://qiita-image-store.s3.ap-northeast-1.amazonaws.com/0/1089224/3d02c021-0d96-2d2d-7759-52d77520cf63.png) |
| measure simple (Euclidean) distance from origin | taxi cab distance| Computationally cheap but grows slowly near origin (not good for distinguishing 0 and non 0) |  For every feature, the minimum value of that feature gets transformed into a 0, the maximum value gets transformed into a 1, and every other value gets transformed into a decimal between 0 and 1 |   1<= p, where p ∈ real number |
| requires ***x*** vector for derivative of element x| | requires its element alone for derivative of element x, i.e., (L^2)^2 == ***X*** T ***X***|  |  |
| ```(25**2 + 2**2 + 5**2)**(1/2) ``` |  | ```(25**2 + 2**2 + 5**2)```|  |  |
| ```np.linalg.norm(x)``` | ```np.abs(25) + np.abs(2) + np.abs(5)```| ```np.dot(x,x)```|```np.max([np.abs(25),np.abs(2),np.abs(5)])``` |  |

- Robustness: L1 > L2
- Stability: L2 > L1
- Solution numeracy: L2 one, L1 many

---
- ※Vector has **magnitude and direction**
- **Unit Vectors**
    - A unit vector is a vector with **magnitude of 1**
    - known as a direction vector
    - notation: a lowercase letter with a cap ('^') symbol

<img width="300" src="https://qiita-image-store.s3.ap-northeast-1.amazonaws.com/0/1089224/2e6447c1-2800-b2c0-45e6-2ec57428e4bb.png">

<img width="200" src="https://qiita-image-store.s3.ap-northeast-1.amazonaws.com/0/1089224/77cc5d26-b855-01fb-77d4-a044a5dd62e5.png">

<img width="500" src="https://qiita-image-store.s3.ap-northeast-1.amazonaws.com/0/1089224/0f4f98ee-de17-a43c-200e-e26f7023d136.png">

<img width="400" src="https://qiita-image-store.s3.ap-northeast-1.amazonaws.com/0/1089224/304feba4-4a59-6d32-881e-1a4f7107ba59.png">


---
### Vector Transposition
  - ```
     x = np.array([[1,2,3]])   // NESTED matrix-style (transposing a 1D has NO effect)
     x_t = x.T
     ```
     <img width="200" src="https://qiita-image-store.s3.ap-northeast-1.amazonaws.com/0/1089224/71b6547f-2f91-75a8-a664-9163b9697cdb.png">
---
### Orthogonal Vectors
- ***x*** and ***y*** are orthogonal vectors if ***x***T***y*** = 0
- ***x*** and ***y*** are at 90 degrees angle to each other 
- n-dimensional space has max n mutually orthogonal vectors
- **Basic vectors are orthogonal**
    - ```
      i = np.array([1,0]) 
      j = np.array([0,1]) 
      np.dot(i,j)  
      //output: 0
      ```

<a id="#Tensor-Operations"></a>
# Tensor Operations
※Tensor = Scalar or ***Vector***
- HADAMARD PRODUCT = ELEMENT-WISE PRODUCT:
     ![image.png](https://qiita-image-store.s3.ap-northeast-1.amazonaws.com/0/1089224/5e8196c4-1cc4-559e-642a-d49c8733bd67.png)  <img width="400" src="https://qiita-image-store.s3.ap-northeast-1.amazonaws.com/0/1089224/3442b26f-6062-de5e-70ad-46c939e57e1b.png">


- DOT PRODUCT
     ![image.png](https://qiita-image-store.s3.ap-northeast-1.amazonaws.com/0/1089224/13ff9fdd-c362-bdb1-e04b-fe862e6630b6.png)    ![image.png](https://qiita-image-store.s3.ap-northeast-1.amazonaws.com/0/1089224/639c4643-8eb5-e16a-bd32-05e09a5c3151.png) 
   <img width="500" src ="https://qiita-image-store.s3.ap-northeast-1.amazonaws.com/0/1089224/7a48ea16-840d-6dd8-d883-be7caf627f63.png">


<a id="#Matrix-Properties"></a>
# Matrix Properties

- FROBENIUS NORM
    - Frobenius norm of a matrix is equal to L2 norm of singular values
    <img width="300" src ="https://qiita-image-store.s3.ap-northeast-1.amazonaws.com/0/1089224/d18b7a0e-2f8f-6443-8d60-7b2e97226a18.png">

   ```
   X = np.array([[1, 2], [3, 4]])
   
   //numpy
   (1**2 + 2**2 + 3**2 + 4**2)**(1/2)
   np.linalg.norm(X)  #same function as for vector L2 norm
   
   //pytorch
   X_pt = torch.tensor([[1, 2], [3, 4.]]) # torch.norm() supports floats only
   torch.norm(X_pt)
   ```

---
- Matrix DOT PRODUCT
      <img width="300" src="https://qiita-image-store.s3.ap-northeast-1.amazonaws.com/0/1089224/c02c214f-8cfd-b3e2-a97b-d63caf2428bd.png">

---
- MATRIX INVERSION 
     <img width="100" src="https://qiita-image-store.s3.ap-northeast-1.amazonaws.com/0/1089224/9df70b6a-acb2-a9e5-c8a2-a7af8afb25f5.png">
    -  ->The inverse of matrix is used of find the solution of linear equations through the matrix inversion method.
    -  e.g., we can find parameters values using matrix inverse below:
          <img width="150" src="https://qiita-image-store.s3.ap-northeast-1.amazonaws.com/0/1089224/d6ff9722-c81e-5597-d220-31ea78a93f29.png">
          then,
          <img width="300" src="https://qiita-image-store.s3.ap-northeast-1.amazonaws.com/0/1089224/2e66c26b-4b5b-0694-48d7-77fe15dc5d46.png">

    - matrix inversion is NOT possible when:
         <img width="700" src="https://qiita-image-store.s3.ap-northeast-1.amazonaws.com/0/1089224/db43cd74-cc12-405c-b5be-2a48d1fc969a.png">


---
- DIAGONAL MATRIX : diag(x) 
    - A square matrix in which every element except the principal diagonal elements is zero
    -  Multiplication: diag(x)y = x ⊙ y : efficient!
    -  Inversion: diag(x)-1 = diag[1/x1, … , 1/xn]T
      <img width="300" src="https://qiita-image-store.s3.ap-northeast-1.amazonaws.com/0/1089224/36dbca29-8de2-15ab-48d4-7ffa2e50a781.png"> 
      <img width="300" src="https://qiita-image-store.s3.ap-northeast-1.amazonaws.com/0/1089224/deac4532-26ba-8575-7ac3-296dd0771aeb.png">
---
- ORTHOGONAL MATRIX
     <img width="300" src="https://qiita-image-store.s3.ap-northeast-1.amazonaws.com/0/1089224/06241c8b-3d17-2cb1-0d20-debe4f264b59.png">  
    - ->Calculating A_transpose is cheap, so if the matrix is orthogonal, use A_inverse!

---
- SYMMETRIC MATRIX
     <img width="500" src="https://qiita-image-store.s3.ap-northeast-1.amazonaws.com/0/1089224/3329ef17-fd29-5271-ff67-420808c58b7b.png">  



<a id="#Eigenvectors-and-Eigenvalues"></a>
# Eigenvectors and Eigenvalues
- 3 options in linear algebra: one, no, or infinite solutions.
- "Eigenvector of a matrix A is a vector represented by a matrix X such that when X is multiplied with matrix A, then the direction of the resultant matrix remains the same as vector X. Mathematically, the above statement can be represented as AX = λX"

-  NumPy eig() method: returns a tuple of a vector of eigenvalues and a matrix of eigenvectors

```
A = np.array([[-1, 4], [2, -2]])

# cheat with NumPy eig() method using eig() which returns touples
Vector_eigenvalues, Matrix_eigenVectors = np.linalg.eig(A)  
#Vector_eigenvaluesas: a vector of eigenvalues
        #Matrix_eigenVectors: a matrix of eigenvectors
```

<img width="500" src="https://qiita-image-store.s3.ap-northeast-1.amazonaws.com/0/1089224/34cb0a29-27ad-fe13-6289-422856b19233.png">


---
### Determinant
 <img width="500" src="https://qiita-image-store.s3.ap-northeast-1.amazonaws.com/0/1089224/480e45d0-0c23-2e6d-214d-8dc562fc36cf.png">

---

### Determinants & Eigenvalues 
- **det(X) = product of all eigenvalues of X**

---

### Matrix Decomposition
- ->
    - ※in order to reconstitute the original matrix
    - Matrix decomposition are mostly used to solve linear systems in a easy way or quickly.


- *Common decomposition methods*
    - **Eigenvalue-based decomposition**
        - mostly applicable to square matrices
        - matrix decomposes into 3 parts: final rotation, scaling, initial rotation
        - "Matrix factorization reduces a computer's storage space for matrices, instead of storing the large non-factorized matrix(A), We can use less storage for its factors, sometimes even smaller when the rank of a matrix is small."



    -  **Singular Value Decomposition (SVD)** 
        -  SVD is a factorization of that matrix into three matrices: U, S, V.
            -  U: left singular valued matrix
            -  S: sigular valued matrix
            -  V: right singular valued matrix.
        - U,V are orthogonal matrices


     <img width="300" src="https://qiita-image-store.s3.ap-northeast-1.amazonaws.com/0/1089224/d07f6ffe-be0d-077f-312a-13e35e43f53d.png">  
     <img width="500" src="https://qiita-image-store.s3.ap-northeast-1.amazonaws.com/0/1089224/98ca436a-ac87-c1d5-e0ad-3bb5ab9de066.png">


<a id="#Matrix-Operations-for-Machine-Learning"></a>
# Matrix Operations for Machine Learning




---




<a id="#limits"></a>
# Limits



<a id="#Derivative-and-Differentiation"></a>
# Derivative and Differentiation


<a id="#Automatic-Differentiation"></a>
# Automatic Differentiation


<a id="#Partial-Derivative-Calculus"></a>
# Partial-Derivative Calculus



<a id="#Integral-Calculus"></a>
# Integral Calculus



<a id="#reference"></a>
# Reference
- https://www.citycollegiate.com/vectorXa.htm
- https://www.w3schools.com/ai/img_tensor.jpg
- https://miro.medium.com/max/1180/1*XmWq6hUA6-FanhFICmMZFg.png
- https://speakerdeck.com/tanimutomo/before-we-begin-the-mathematical-building-blocks-of-neural-networks?slide=39
- https://www.quora.com/How-can-I-calculate-the-inverse-of-a-diagonal-matrix
- https://www.geeksforgeeks.org/how-to-calculate-the-unit-vector/
- https://byjus.com/questions/what-is-unit-vector-in-physics/
- https://www.geeksforgeeks.org/eigen-values-and-eigen-vectors/
- https://medium.com/analytics-vidhya/understanding-of-matrix-factorization-mf-and-singular-value-decomposition-svd-1a38c2d5bbaa
-