Evolutionary Structure Learning
for Tensor Network Decomposition
Chao Li
Postdoc researcher at TLTRIKEN-AIP
chao.li@riken.jp

Learning with Bigger Data
‣ Bigger data always imply we can learn more knowledge.
‣ High-dimensional, multi-modal, and incomplete‣ Big data need bigger models: Over-parameterization

Deep neural network

How to efficiently represent a HD problem with few parameters?
2

Our Solution: Tensor Network (TN)‣ TN is not a learning model but an efficient model representation.
‣ TNs efficiently exploit the low-rank structures of the model.
‣ There are lots of theoretical results to analyze the performance.
Data/Model

Restoration

Tensorization

Decomposition

Tensor network is a simple yet promising framework.
3

From Topology Search to Structure Learning
Evolutionary Topology Search for Tensor Network
Decomposition: an Extended Abstract
Chao Li1 , Zhun Sun1 , Junhua Zeng1,2 , and Qibin Zhao1⇤
1
RIKEN Center for Advanced Intelligence Project (AIP), Tokyo, Japan
2
Guangdong University of Technology{chao.li, qibin.zhao}@riken.jp

Abstract
Tensor network (TN) decomposition is a promising framework to represent extremely high-dimensional problems with few parameters. However, it remains
challenging to search the (near-)optimal topological structures for TN decomposition, since the number of candidate solutions exponentially grows with increasing
the order of a tensor. In this work, we claim that the issue can be practicallytackled by genetic algorithms (GAs) in an affordable manner, and the key is to
encode the complex topological structures into fixed-length binary strings, a.k.a.,chromosomes in the context of GA. The experimental results on natural images
demonstrate that, in the decomposition task, GA is able to discover more efficient
topologys than the well-known TN models within a small number of generations.
Our code is available at https://github.com/minogame/icml2020-TNGA.

[Li and Sun, ICML,
2020; Li et al., Neurips workshop on QTNML, 2020]
1 Introduction

4Outline‣ Preliminaries: Tensor network and diagram notation‣ Structure Learning of Tensor Network Decomposition‣ Tensor Network Decomposition Meets Genetic Algorithm

5Part 1:
Tensor and Tensor Network

Tensor is Ubiquitous
Intuitively, TENSOR is number-block.

Vectors and matrices

Tensor cubic

Scalar:Vector:Matrix:Tensor:Numberblocks is a British children's CGI-animated TV series.

Order-0
TensorscalarOrder-1
TensorvectorOrder-2
TensormatrixOrder-3
TensorOrder-5
TensorDiagram NotationJ<latexit sha1_base64="iPKhNFSPiDW86sRcCxgyDM+uv8o=">AAAB93icbVA9SwNBEN2LXzF+RS1tFoNgFe6ioGXQRqwSMB+QHGFvM5cs2d07dveE48gvsNXaTmz9OZb+EzfJFSbxwcDjvRlm5gUxZ9q47rdT2Njc2t4p7pb29g8Oj8rHJ20dJYpCi0Y8Ut2AaOBMQssww6EbKyAi4NAJJvczv/MMSrNIPpk0Bl+QkWQho8RYqfk4KFfcqjsHXideTiooR2NQ/ukPI5oIkIZyonXPc2PjZ0QZRjlMS/1EQ0zohIygZ6kkArSfzQ+d4gurDHEYKVvS4Ln6dyIjQutUBLZTEDPWq95M/M/rJSa89TMm48SApItFYcKxifDsazxkCqjhqSWEKmZvxXRMFKHGZrO0JRBTm4m3msA6adeq3lW11ryu1O/ydIroDJ2jS+ShG1RHD6iBWogiQC/oFb05qfPufDifi9aCk8+coiU4X79ELZNo</latexit>II(1 ⇥ 1)(I ⇥ 1)(I ⇥ J)

ScalarVectorMatrix<latexit sha1_base64="6MgDcebxBw4AHPTGRkLXsNY4hn8=">AAAB93icbVA9SwNBEN2LXzF+RS1tFoNgFe6ioGXQRrsEzAckR9jbzCVLdveO3T3hOPILbLW2E1t/jqX/xE1yhUl8MPB4b4aZeUHMmTau++0UNja3tneKu6W9/YPDo/LxSVtHiaLQohGPVDcgGjiT0DLMcOjGCogIOHSCyf3M7zyD0iySTyaNwRdkJFnIKDFWaj4OyhW36s6B14mXkwrK0RiUf/rDiCYCpKGcaN3z3Nj4GVGGUQ7TUj/REBM6ISPoWSqJAO1n80On+MIqQxxGypY0eK7+nciI0DoVge0UxIz1qjcT//N6iQlv/YzJODEg6WJRmHBsIjz7Gg+ZAmp4agmhitlbMR0TRaix2SxtCcTUZuKtJrBO2rWqd1WtNa8r9bs8nSI6Q+foEnnoBtXRA2qgFqII0At6RW9O6rw7H87norXg5DOnaAnO1y9CmpNn</latexit><latexit sha1_base64="EZEmgoPYGnSmLv9tAdgJRCcw6xM=">AAACAXicbVDLSgMxFM34rPVVdekmWIS6KTNV0GXRjcsK9gHToWTSTBuaTIbkjlCGrvwGt7p2J279Epf+iWk7C9t6IHA4517OzQkTwQ247reztr6xubVd2Cnu7u0fHJaOjltGpZqyJlVC6U5IDBM8Zk3gIFgn0YzIULB2OLqb+u0npg1X8SOMExZIMoh5xCkBK/kVrwtcMoO9i16p7FbdGfAq8XJSRjkavdJPt69oKlkMVBBjfM9NIMiIBk4FmxS7qWEJoSMyYL6lMbE5QTY7eYLPrdLHkdL2xYBn6t+NjEhjxjK0k5LA0Cx7U/E/z08hugkyHicpsJjOg6JUYFB4+n/c55pREGNLCNXc3orpkGhCwba0kBLKie3EW25glbRqVe+yWnu4Ktdv83YK6BSdoQry0DWqo3vUQE1EkUIv6BW9Oc/Ou/PhfM5H15x85wQtwPn6BYuultM=</latexit><latexit sha1_base64="6MgDcebxBw4AHPTGRkLXsNY4hn8=">AAAB93icbVA9SwNBEN2LXzF+RS1tFoNgFe6ioGXQRrsEzAckR9jbzCVLdveO3T3hOPILbLW2E1t/jqX/xE1yhUl8MPB4b4aZeUHMmTau++0UNja3tneKu6W9/YPDo/LxSVtHiaLQohGPVDcgGjiT0DLMcOjGCogIOHSCyf3M7zyD0iySTyaNwRdkJFnIKDFWaj4OyhW36s6B14mXkwrK0RiUf/rDiCYCpKGcaN3z3Nj4GVGGUQ7TUj/REBM6ISPoWSqJAO1n80On+MIqQxxGypY0eK7+nciI0DoVge0UxIz1qjcT//N6iQlv/YzJODEg6WJRmHBsIjz7Gg+ZAmp4agmhitlbMR0TRaix2SxtCcTUZuKtJrBO2rWqd1WtNa8r9bs8nSI6Q+foEnnoBtXRA2qgFqII0At6RW9O6rw7H87norXg5DOnaAnO1y9CmpNn</latexit><latexit sha1_base64="CdDEFOmD07BOoEI/ujiP3GVwNE8=">AAACAXicbVDLSgMxFL1TX7W+qi7dBItQN2WmCrosutFdBfuA6VAyaaYNTTJDkhFK6cpvcKtrd+LWL3Hpn5i2s7CtBwKHc+7l3Jww4Uwb1/12cmvrG5tb+e3Czu7e/kHx8Kip41QR2iAxj1U7xJpyJmnDMMNpO1EUi5DTVji8nfqtJ6o0i+WjGSU0ELgvWcQINlbyy/cdwwTVyDvvFktuxZ0BrRIvIyXIUO8Wfzq9mKSCSkM41tr33MQEY6wMI5xOCp1U0wSTIe5T31KJbU4wnp08QWdW6aEoVvZJg2bq340xFlqPRGgnBTYDvexNxf88PzXRdTBmMkkNlWQeFKUcmRhN/496TFFi+MgSTBSztyIywAoTY1taSAnFxHbiLTewSprVindRqT5clmo3WTt5OIFTKIMHV1CDO6hDAwjE8AKv8OY8O+/Oh/M5H8052c4xLMD5+gWyTpbr</latexit><latexit sha1_base64="iPTNFxUONcbVk7ikbpNWDPrDMUc=">AAACAXicbVDLSgMxFM34rPVVdekmWIS6KTNV0GXRjbqqYB8wHUomTdvQZDIkd4QydOU3uNW1O3Hrl7j0T0zbWdjWA4HDOfdybk4YC27Adb+dldW19Y3N3FZ+e2d3b79wcNgwKtGU1akSSrdCYpjgEasDB8FasWZEhoI1w+HNxG8+MW24ih5hFLNAkn7Ee5wSsJJfumsDl8zg+7NOoeiW3SnwMvEyUkQZap3CT7uraCJZBFQQY3zPjSFIiQZOBRvn24lhMaFD0me+pRGxOUE6PXmMT63SxT2l7YsAT9W/GymRxoxkaCclgYFZ9Cbif56fQO8qSHkUJ8AiOgvqJQKDwpP/4y7XjIIYWUKo5vZWTAdEEwq2pbmUUI5tJ95iA8ukUSl75+XKw0Wxep21k0PH6ASVkIcuURXdohqqI4oUekGv6M15dt6dD+dzNrriZDtHaA7O1y/ZwpcE</latexit>IN<latexit sha1_base64="EveJdKz84u2uWiserDCk67TI7c4=">AAAB+XicbVA9SwNBEJ2LXzF+RS1tFoNgFe5iQMugjTYS0XxAcoS9zV6yZHfv2N0TwpGfYKu1ndj6ayz9J26SK0zig4HHezPMzAtizrRx3W8nt7a+sbmV3y7s7O7tHxQPj5o6ShShDRLxSLUDrClnkjYMM5y2Y0WxCDhtBaObqd96pkqzSD6ZcUx9gQeShYxgY6XHu959r1hyy+4MaJV4GSlBhnqv+NPtRyQRVBrCsdYdz42Nn2JlGOF0UugmmsaYjPCAdiyVWFDtp7NTJ+jMKn0URsqWNGim/p1IsdB6LALbKbAZ6mVvKv7ndRITXvkpk3FiqCTzRWHCkYnQ9G/UZ4oSw8eWYKKYvRWRIVaYGJvOwpZATGwm3nICq6RZKXsX5cpDtVS7ztLJwwmcwjl4cAk1uIU6NIDAAF7gFd6c1Hl3PpzPeWvOyWaOYQHO1y+d2ZQo</latexit>

K

J<latexit sha1_base64="yQchR/vGmI1FWVj1ElrvAcWIZUI=">AAAB93icbVA9SwNBEN2LXzF+RS1tFoNgFe6ioGXQRrBJwHxAcoS9zVyyZHfv2N0TjiO/wFZrO7H151j6T9wkV5jEBwOP92aYmRfEnGnjut9OYWNza3unuFva2z84PCofn7R1lCgKLRrxSHUDooEzCS3DDIdurICIgEMnmNzP/M4zKM0i+WTSGHxBRpKFjBJjpebjoFxxq+4ceJ14OamgHI1B+ac/jGgiQBrKidY9z42NnxFlGOUwLfUTDTGhEzKCnqWSCNB+Nj90ii+sMsRhpGxJg+fq34mMCK1TEdhOQcxYr3oz8T+vl5jw1s+YjBMDki4WhQnHJsKzr/GQKaCGp5YQqpi9FdMxUYQam83SlkBMbSbeagLrpF2relfVWvO6Ur/L0ymiM3SOLpGHblAdPaAGaiGKAL2gV/TmpM678+F8LloLTj5zipbgfP0CRcCTaQ==</latexit>

Ik<latexit sha1_base64="iPKhNFSPiDW86sRcCxgyDM+uv8o=">AAAB93icbVA9SwNBEN2LXzF+RS1tFoNgFe6ioGXQRqwSMB+QHGFvM5cs2d07dveE48gvsNXaTmz9OZb+EzfJFSbxwcDjvRlm5gUxZ9q47rdT2Njc2t4p7pb29g8Oj8rHJ20dJYpCi0Y8Ut2AaOBMQssww6EbKyAi4NAJJvczv/MMSrNIPpk0Bl+QkWQho8RYqfk4KFfcqjsHXideTiooR2NQ/ukPI5oIkIZyonXPc2PjZ0QZRjlMS/1EQ0zohIygZ6kkArSfzQ+d4gurDHEYKVvS4Ln6dyIjQutUBLZTEDPWq95M/M/rJSa89TMm48SApItFYcKxifDsazxkCqjhqSWEKmZvxXRMFKHGZrO0JRBTm4m3msA6adeq3lW11ryu1O/ydIroDJ2jS+ShG1RHD6iBWogiQC/oFb05qfPufDifi9aCk8+coiU4X79ELZNo</latexit>

I<latexit sha1_base64="6MgDcebxBw4AHPTGRkLXsNY4hn8=">AAAB93icbVA9SwNBEN2LXzF+RS1tFoNgFe6ioGXQRrsEzAckR9jbzCVLdveO3T3hOPILbLW2E1t/jqX/xE1yhUl8MPB4b4aZeUHMmTau++0UNja3tneKu6W9/YPDo/LxSVtHiaLQohGPVDcgGjiT0DLMcOjGCogIOHSCyf3M7zyD0iySTyaNwRdkJFnIKDFWaj4OyhW36s6B14mXkwrK0RiUf/rDiCYCpKGcaN3z3Nj4GVGGUQ7TUj/REBM6ISPoWSqJAO1n80On+MIqQxxGypY0eK7+nciI0DoVge0UxIz1qjcT//N6iQlv/YzJODEg6WJRmHBsIjz7Gg+ZAmp4agmhitlbMR0TRaix2SxtCcTUZuKtJrBO2rWqd1WtNa8r9bs8nSI6Q+foEnnoBtXRA2qgFqII0At6RW9O6rw7H87norXg5DOnaAnO1y9CmpNn</latexit><latexit sha1_base64="dnHEp4lgdA4li6dUWQoii/vavnY=">AAAB+XicbVA9SwNBEJ2LXzF+RS1tFoNgFe6ioGXQRruI5gOSI+xt9pIlu3vH7p4QjvwEW63txNZfY+k/cS+5wiQ+GHi8N8PMvCDmTBvX/XYKa+sbm1vF7dLO7t7+QfnwqKWjRBHaJBGPVCfAmnImadMww2knVhSLgNN2ML7N/PYzVZpF8slMYuoLPJQsZAQbKz3e98f9csWtujOgVeLlpAI5Gv3yT28QkURQaQjHWnc9NzZ+ipVhhNNpqZdoGmMyxkPatVRiQbWfzk6dojOrDFAYKVvSoJn6dyLFQuuJCGynwGakl71M/M/rJia89lMm48RQSeaLwoQjE6HsbzRgihLDJ5Zgopi9FZERVpgYm87ClkBMbSbecgKrpFWrehfV2sNlpX6Tp1OEEziFc/DgCupwBw1oAoEhvMArvDmp8+58OJ/z1oKTzxzDApyvX8uAlEU=</latexit>

I1

I1

IN

<latexit sha1_base64="O/m4HxTLUmr6XSW2cPgdyWfZvck=">AAAB+XicbVA9SwNBEJ2LXzF+RS1tFoNgFe6ioGXQRruI5gOSI+xt9pIlu3vH7p4QjvwEW63txNZfY+k/cS+5wiQ+GHi8N8PMvCDmTBvX/XYKa+sbm1vF7dLO7t7+QfnwqKWjRBHaJBGPVCfAmnImadMww2knVhSLgNN2ML7N/PYzVZpF8slMYuoLPJQsZAQbKz3e971+ueJW3RnQKvFyUoEcjX75pzeISCKoNIRjrbueGxs/xcowwum01Es0jTEZ4yHtWiqxoNpPZ6dO0ZlVBiiMlC1p0Ez9O5FiofVEBLZTYDPSy14m/ud1ExNe+ymTcWKoJPNFYcKRiVD2NxowRYnhE0swUczeisgIK0yMTWdhSyCmNhNvOYFV0qpVvYtq7eGyUr/J0ynCCZzCOXhwBXW4gwY0gcAQXuAV3pzUeXc+nM95a8HJZ45hAc7XL3AylAs=</latexit><latexit sha1_base64="EveJdKz84u2uWiserDCk67TI7c4=">AAAB+XicbVA9SwNBEJ2LXzF+RS1tFoNgFe5iQMugjTYS0XxAcoS9zV6yZHfv2N0TwpGfYKu1ndj6ayz9J26SK0zig4HHezPMzAtizrRx3W8nt7a+sbmV3y7s7O7tHxQPj5o6ShShDRLxSLUDrClnkjYMM5y2Y0WxCDhtBaObqd96pkqzSD6ZcUx9gQeShYxgY6XHu959r1hyy+4MaJV4GSlBhnqv+NPtRyQRVBrCsdYdz42Nn2JlGOF0UugmmsaYjPCAdiyVWFDtp7NTJ+jMKn0URsqWNGim/p1IsdB6LALbKbAZ6mVvKv7ndRITXvkpk3FiqCTzRWHCkYnQ9G/UZ4oSw8eWYKKYvRWRIVaYGJvOwpZATGwm3nICq6RZKXsX5cpDtVS7ztLJwwmcwjl4cAk1uIU6NIDAAF7gFd6c1Hl3PpzPeWvOyWaOYQHO1y+d2ZQo</latexit><latexit sha1_base64="O/m4HxTLUmr6XSW2cPgdyWfZvck=">AAAB+XicbVA9SwNBEJ2LXzF+RS1tFoNgFe6ioGXQRruI5gOSI+xt9pIlu3vH7p4QjvwEW63txNZfY+k/cS+5wiQ+GHi8N8PMvCDmTBvX/XYKa+sbm1vF7dLO7t7+QfnwqKWjRBHaJBGPVCfAmnImadMww2knVhSLgNN2ML7N/PYzVZpF8slMYuoLPJQsZAQbKz3e971+ueJW3RnQKvFyUoEcjX75pzeISCKoNIRjrbueGxs/xcowwum01Es0jTEZ4yHtWiqxoNpPZ6dO0ZlVBiiMlC1p0Ez9O5FiofVEBLZTYDPSy14m/ud1ExNe+ymTcWKoJPNFYcKRiVD2NxowRYnhE0swUczeisgIK0yMTWdhSyCmNhNvOYFV0qpVvYtq7eGyUr/J0ynCCZzCOXhwBXW4gwY0gcAQXuAV3pzUeXc+nM95a8HJZ45hAc7XL3AylAs=</latexit>N<latexit sha1_base64="39NGy8O4iwzC1Rc29vY7F6jZCrU=">AAAB93icbVA9SwNBEN2LXzF+RS1tFoNgFe6ioGXQxkoSMB+QHGFvM5cs2d07dveE48gvsNXaTmz9OZb+EzfJFSbxwcDjvRlm5gUxZ9q47rdT2Njc2t4p7pb29g8Oj8rHJ20dJYpCi0Y8Ut2AaOBMQssww6EbKyAi4NAJJvczv/MMSrNIPpk0Bl+QkWQho8RYqfk4KFfcqjsHXideTiooR2NQ/ukPI5oIkIZyonXPc2PjZ0QZRjlMS/1EQ0zohIygZ6kkArSfzQ+d4gurDHEYKVvS4Ln6dyIjQutUBLZTEDPWq95M/M/rJSa89TMm48SApItFYcKxifDsazxkCqjhqSWEKmZvxXRMFKHGZrO0JRBTm4m3msA6adeq3lW11ryu1O/ydIroDJ2jS+ShG1RHD6iBWogiQC/oFb05qfPufDifi9aCk8+coiU4X79KeZNs</latexit>(I ⇥ J ⇥ K)

(I1 ⇥ I2 ⇥ · · · ⇥ IN )order-3 Tensor

order-N Tensor<latexit sha1_base64="2dY5dknUY7GVUPyF4XCBF6lv4hM=">AAACC3icbVDLSgMxFM34rPU12qWbYBHqpsxUQZdFNz42FewD2qFk0kwbmskMyR1hKP0Ev8Gtrt2JWz/CpX9i2s7Cth4IOZxzL+dy/FhwDY7zba2srq1vbOa28ts7u3v79sFhQ0eJoqxOIxGplk80E1yyOnAQrBUrRkJfsKY/vJ74zSemNI/kI6Qx80LSlzzglICRunahdNsBHjKN77L//rRrF52yMwVeJm5GiihDrWv/dHoRTUImgQqiddt1YvBGRAGngo3znUSzmNAh6bO2oZKYHG80PX6MT4zSw0GkzJOAp+rfjREJtU5D30yGBAZ60ZuI/3ntBIJLb8RlnACTdBYUJAJDhCdN4B5XjIJIDSFUcXMrpgOiCAXT11yKH45NJ+5iA8ukUSm7Z+XKw3mxepW1k0NH6BiVkIsuUBXdoBqqI4pS9IJe0Zv1bL1bH9bnbHTFynYKaA7W1y86S5pu</latexit><latexit sha1_base64="RlY4qj8aJU3bHXR0cym/vC92qLw=">AAACHXicbVDLSgMxFM3UV62vUZdugqVQN2WmCrosurEbqWAf0JYhk8m0oZnMkNwRSukX+BF+g1tduxO34tI/MX0g2nogcDjnXs7N8RPBNTjOp5VZWV1b38hu5ra2d3b37P2Dho5TRVmdxiJWLZ9oJrhkdeAgWCtRjES+YE1/cDXxm/dMaR7LOxgmrBuRnuQhpwSM5NmFYtVzO8AjpnHVK89YhwYx6B/15sSz807JmQIvE3dO8miOmmd/dYKYphGTQAXRuu06CXRHRAGngo1znVSzhNAB6bG2oZKYpO5o+p0xLhglwGGszJOAp+rvjRGJtB5GvpmMCPT1ojcR//PaKYQX3RGXSQpM0llQmAoMMZ50gwOuGAUxNIRQxc2tmPaJIhRMg39S/GhsOnEXG1gmjXLJPS2Vb8/ylct5O1l0hI5REbnoHFXQNaqhOqLoAT2hZ/RiPVqv1pv1PhvNWPOdQ/QH1sc3LISh5A==</latexit>

8Tensor ContractionabA<latexit sha1_base64="94tL1q9O9cS77J172E010NqOwHQ=">AAAB/3icbVC7TsMwFL3hWcqrwMhiUSExVUlBgrHAwlgk+kBtVDmu01q1nch2kKqoA9/ACjMbYuVTGPkTnDYDbTmSpaNz7tU9PkHMmTau++2srK6tb2wWtorbO7t7+6WDw6aOEkVog0Q8Uu0Aa8qZpA3DDKftWFEsAk5bweg281tPVGkWyQczjqkv8ECykBFsrPTYFdgMgxBd90plt+JOgZaJl5My5Kj3Sj/dfkQSQaUhHGvd8dzY+ClWhhFOJ8VuommMyQgPaMdSiQXVfjoNPEGnVumjMFL2SYOm6t+NFAutxyKwk1lAvehl4n9eJzHhlZ8yGSeGSjI7FCYcmQhlv0d9pigxfGwJJorZrIgMscLE2I7mrgRiYjvxFhtYJs1qxTuvVO8vyrWbvJ0CHMMJnIEHl1CDO6hDAwgIeIFXeHOenXfnw/mcja44+c4RzMH5+gUI7Zad</latexit>

T

a b

I
<latexit sha1_base64="6MgDcebxBw4AHPTGRkLXsNY4hn8=">AAAB93icbVA9SwNBEN2LXzF+RS1tFoNgFe6ioGXQRrsEzAckR9jbzCVLdveO3T3hOPILbLW2E1t/jqX/xE1yhUl8MPB4b4aZeUHMmTau++0UNja3tneKu6W9/YPDo/LxSVtHiaLQohGPVDcgGjiT0DLMcOjGCogIOHSCyf3M7zyD0iySTyaNwRdkJFnIKDFWaj4OyhW36s6B14mXkwrK0RiUf/rDiCYCpKGcaN3z3Nj4GVGGUQ7TUj/REBM6ISPoWSqJAO1n80On+MIqQxxGypY0eK7+nciI0DoVge0UxIz1qjcT//N6iQlv/YzJODEg6WJRmHBsIjz7Gg+ZAmp4agmhitlbMR0TRaix2SxtCcTUZuKtJrBO2rWqd1WtNa8r9bs8nSI6Q+foEnnoBtXRA2qgFqII0At6RW9O6rw7H87norXg5DOnaAnO1y9CmpNn</latexit>

AAb<latexit sha1_base64="iPKhNFSPiDW86sRcCxgyDM+uv8o=">AAAB93icbVA9SwNBEN2LXzF+RS1tFoNgFe6ioGXQRqwSMB+QHGFvM5cs2d07dveE48gvsNXaTmz9OZb+EzfJFSbxwcDjvRlm5gUxZ9q47rdT2Njc2t4p7pb29g8Oj8rHJ20dJYpCi0Y8Ut2AaOBMQssww6EbKyAi4NAJJvczv/MMSrNIPpk0Bl+QkWQho8RYqfk4KFfcqjsHXideTiooR2NQ/ukPI5oIkIZyonXPc2PjZ0QZRjlMS/1EQ0zohIygZ6kkArSfzQ+d4gurDHEYKVvS4Ln6dyIjQutUBLZTEDPWq95M/M/rJSa89TMm48SApItFYcKxifDsazxkCqjhqSWEKmZvxXRMFKHGZrO0JRBTm4m3msA6adeq3lW11ryu1O/ydIroDJ2jS+ShG1RHD6iBWogiQC/oFb05qfPufDifi9aCk8+coiU4X79ELZNo</latexit>(1 ⇥ 1)<latexit sha1_base64="DNGegz3UpEy/SoVyc/0rjZStnVY=">AAAB/3icbVDLSgMxFL1TX7W+qi7dBIvgqsxUQZdFNy4r2Ie0Q0nSTBuazAxJRihDF36DW127E7d+ikv/xEw7C9t6IHA4517uySGx4Nq47rdTWFvf2Nwqbpd2dvf2D8qHRy0dJYqyJo1EpDoEayZ4yJqGG8E6sWJYEsHaZHyb+e0npjSPwgcziZkv8TDkAafYWOmxJ7EZkQCRfrniVt0Z0CrxclKBHI1++ac3iGgiWWiowFp3PTc2foqV4VSwaamXaBZjOsZD1rU0xJJpP50FnqIzqwxQECn7QoNm6t+NFEutJ5LYySygXvYy8T+vm5jg2k95GCeGhXR+KEgEMhHKfo8GXDFqxMQSTBW3WREdYYWpsR0tXCFyajvxlhtYJa1a1buo1u4vK/WbvJ0inMApnIMHV1CHO2hAEyhIeIFXeHOenXfnw/mcjxacfOcYFuB8/QI84Ja+</latexit>ab<latexit sha1_base64="6MgDcebxBw4AHPTGRkLXsNY4hn8=">AAAB93icbVA9SwNBEN2LXzF+RS1tFoNgFe6ioGXQRrsEzAckR9jbzCVLdveO3T3hOPILbLW2E1t/jqX/xE1yhUl8MPB4b4aZeUHMmTau++0UNja3tneKu6W9/YPDo/LxSVtHiaLQohGPVDcgGjiT0DLMcOjGCogIOHSCyf3M7zyD0iySTyaNwRdkJFnIKDFWaj4OyhW36s6B14mXkwrK0RiUf/rDiCYCpKGcaN3z3Nj4GVGGUQ7TUj/REBM6ISPoWSqJAO1n80On+MIqQxxGypY0eK7+nciI0DoVge0UxIz1qjcT//N6iQlv/YzJODEg6WJRmHBsIjz7Gg+ZAmp4agmhitlbMR0TRaix2SxtCcTUZuKtJrBO2rWqd1WtNa8r9bs8nSI6Q+foEnnoBtXRA2qgFqII0At6RW9O6rw7H87norXg5DOnaAnO1y9CmpNn</latexit>cX<latexit sha1_base64="KRj5Yh3W8hsYBioRpujyXxO60FU=">AAAB/3icbVDLSgMxFL1TX7W+qi7dBIvgqsxUQZdFNy4r2Ie0Q8mkmTY0yQxJRihDF36DW127E7d+ikv/xEw7C9t6IHA4517uyQlizrRx3W+nsLa+sblV3C7t7O7tH5QPj1o6ShShTRLxSHUCrClnkjYNM5x2YkWxCDhtB+PbzG8/UaVZJB/MJKa+wEPJQkawsdJjT2AzCkJE+uWKW3VnQKvEy0kFcjT65Z/eICKJoNIQjrXuem5s/BQrwwin01Iv0TTGZIyHtGupxIJqP50FnqIzqwxQGCn7pEEz9e9GioXWExHYySygXvYy8T+vm5jw2k+ZjBNDJZkfChOOTISy36MBU5QYPrEEE8VsVkRGWGFibEcLVwIxtZ14yw2sklat6l1Ua/eXlfpN3k4RTuAUzsGDK6jDHTSgCQQEvMArvDnPzrvz4XzORwtOvnMMC3C+fgE+c5a/</latexit><latexit sha1_base64="gJU+xKS8gcd+TGzfn1VAZmamC10=">AAACAXicbVDLSgMxFL3js9ZX1aWbYBFclZkq6LLoxmUF+4DpUDJppg1NMkOSEcrQld/gVtfuxK1f4tI/MdPOwrYeCBzOuZd7csKEM21c99tZW9/Y3Nou7ZR39/YPDitHx20dp4rQFol5rLoh1pQzSVuGGU67iaJYhJx2wvFd7neeqNIslo9mktBA4KFkESPYWMnvCWxGBPOsO+1Xqm7NnQGtEq8gVSjQ7Fd+eoOYpIJKQzjW2vfcxAQZVoYRTqflXqppgskYD6lvqcSC6iCbRZ6ic6sMUBQr+6RBM/XvRoaF1hMR2sk8ol72cvE/z09NdBNkTCapoZLMD0UpRyZG+f/RgClKDJ9YgoliNisiI6wwMbalhSuhyDvxlhtYJe16zbus1R+uqo3bop0SnMIZXIAH19CAe2hCCwjE8AKv8OY8O+/Oh/M5H11zip0TWIDz9Qt3B5gI</latexit>

K<latexit sha1_base64="yQchR/vGmI1FWVj1ElrvAcWIZUI=">AAAB93icbVA9SwNBEN2LXzF+RS1tFoNgFe6ioGXQRrBJwHxAcoS9zVyyZHfv2N0TjiO/wFZrO7H151j6T9wkV5jEBwOP92aYmRfEnGnjut9OYWNza3unuFva2z84PCofn7R1lCgKLRrxSHUDooEzCS3DDIdurICIgEMnmNzP/M4zKM0i+WTSGHxBRpKFjBJjpebjoFxxq+4ceJ14OamgHI1B+ac/jGgiQBrKidY9z42NnxFlGOUwLfUTDTGhEzKCnqWSCNB+Nj90ii+sMsRhpGxJg+fq34mMCK1TEdhOQcxYr3oz8T+vl5jw1s+YjBMDki4WhQnHJsKzr/GQKaCGp5YQqpi9FdMxUYQam83SlkBMbSbeagLrpF2relfVWvO6Ur/L0ymiM3SOLpGHblAdPaAGaiGKAL2gV/TmpM678+F8LloLTj5zipbgfP0CRcCTaQ==</latexit>

I<latexit sha1_base64="6MgDcebxBw4AHPTGRkLXsNY4hn8=">AAAB93icbVA9SwNBEN2LXzF+RS1tFoNgFe6ioGXQRrsEzAckR9jbzCVLdveO3T3hOPILbLW2E1t/jqX/xE1yhUl8MPB4b4aZeUHMmTau++0UNja3tneKu6W9/YPDo/LxSVtHiaLQohGPVDcgGjiT0DLMcOjGCogIOHSCyf3M7zyD0iySTyaNwRdkJFnIKDFWaj4OyhW36s6B14mXkwrK0RiUf/rDiCYCpKGcaN3z3Nj4GVGGUQ7TUj/REBM6ISPoWSqJAO1n80On+MIqQxxGypY0eK7+nciI0DoVge0UxIz1qjcT//N6iQlv/YzJODEg6WJRmHBsIjz7Gg+ZAmp4agmhitlbMR0TRaix2SxtCcTUZuKtJrBO2rWqd1WtNa8r9bs8nSI6Q+foEnnoBtXRA2qgFqII0At6RW9O6rw7H87norXg5DOnaAnO1y9CmpNn</latexit>

J<latexit sha1_base64="iPKhNFSPiDW86sRcCxgyDM+uv8o=">AAAB93icbVA9SwNBEN2LXzF+RS1tFoNgFe6ioGXQRqwSMB+QHGFvM5cs2d07dveE48gvsNXaTmz9OZb+EzfJFSbxwcDjvRlm5gUxZ9q47rdT2Njc2t4p7pb29g8Oj8rHJ20dJYpCi0Y8Ut2AaOBMQssww6EbKyAi4NAJJvczv/MMSrNIPpk0Bl+QkWQho8RYqfk4KFfcqjsHXideTiooR2NQ/ukPI5oIkIZyonXPc2PjZ0QZRjlMS/1EQ0zohIygZ6kkArSfzQ+d4gurDHEYKVvS4Ln6dyIjQutUBLZTEDPWq95M/M/rJSa89TMm48SApItFYcKxifDsazxkCqjhqSWEKmZvxXRMFKHGZrO0JRBTm4m3msA6adeq3lW11ryu1O/ydIroDJ2jS+ShG1RHD6iBWogiQC/oFb05qfPufDifi9aCk8+coiU4X79ELZNo</latexit><latexit sha1_base64="iPTNFxUONcbVk7ikbpNWDPrDMUc=">AAACAXicbVDLSgMxFM34rPVVdekmWIS6KTNV0GXRjbqqYB8wHUomTdvQZDIkd4QydOU3uNW1O3Hrl7j0T0zbWdjWA4HDOfdybk4YC27Adb+dldW19Y3N3FZ+e2d3b79wcNgwKtGU1akSSrdCYpjgEasDB8FasWZEhoI1w+HNxG8+MW24ih5hFLNAkn7Ee5wSsJJfumsDl8zg+7NOoeiW3SnwMvEyUkQZap3CT7uraCJZBFQQY3zPjSFIiQZOBRvn24lhMaFD0me+pRGxOUE6PXmMT63SxT2l7YsAT9W/GymRxoxkaCclgYFZ9Cbif56fQO8qSHkUJ8AiOgvqJQKDwpP/4y7XjIIYWUKo5vZWTAdEEwq2pbmUUI5tJ95iA8ukUSl75+XKw0Wxep21k0PH6ASVkIcuURXdohqqI4oUekGv6M15dt6dD+dzNrriZDtHaA7O1y/ZwpcE</latexit><latexit sha1_base64="yQchR/vGmI1FWVj1ElrvAcWIZUI=">AAAB93icbVA9SwNBEN2LXzF+RS1tFoNgFe6ioGXQRrBJwHxAcoS9zVyyZHfv2N0TjiO/wFZrO7H151j6T9wkV5jEBwOP92aYmRfEnGnjut9OYWNza3unuFva2z84PCofn7R1lCgKLRrxSHUDooEzCS3DDIdurICIgEMnmNzP/M4zKM0i+WTSGHxBRpKFjBJjpebjoFxxq+4ceJ14OamgHI1B+ac/jGgiQBrKidY9z42NnxFlGOUwLfUTDTGhEzKCnqWSCNB+Nj90ii+sMsRhpGxJg+fq34mMCK1TEdhOQcxYr3oz8T+vl5jw1s+YjBMDki4WhQnHJsKzr/GQKaCGp5YQqpi9FdMxUYQam83SlkBMbSbeagLrpF2relfVWvO6Ur/L0ymiM3SOLpGHblAdPaAGaiGKAL2gV/TmpM678+F8LloLTj5zipbgfP0CRcCTaQ==</latexit><latexit sha1_base64="6MgDcebxBw4AHPTGRkLXsNY4hn8=">AAAB93icbVA9SwNBEN2LXzF+RS1tFoNgFe6ioGXQRrsEzAckR9jbzCVLdveO3T3hOPILbLW2E1t/jqX/xE1yhUl8MPB4b4aZeUHMmTau++0UNja3tneKu6W9/YPDo/LxSVtHiaLQohGPVDcgGjiT0DLMcOjGCogIOHSCyf3M7zyD0iySTyaNwRdkJFnIKDFWaj4OyhW36s6B14mXkwrK0RiUf/rDiCYCpKGcaN3z3Nj4GVGGUQ7TUj/REBM6ISPoWSqJAO1n80On+MIqQxxGypY0eK7+nciI0DoVge0UxIz1qjcT//N6iQlv/YzJODEg6WJRmHBsIjz7Gg+ZAmp4agmhitlbMR0TRaix2SxtCcTUZuKtJrBO2rWqd1WtNa8r9bs8nSI6Q+foEnnoBtXRA2qgFqII0At6RW9O6rw7H87norXg5DOnaAnO1y9CmpNn</latexit>b(I ⇥ K)<latexit sha1_base64="M//2C6WLxtNTfHKeYcKIzygckwU=">AAACAXicbVDLSgMxFM34rPVVdekmWIS6KTNV0GXRjeKmgn3AdCiZNG1Dk8mQ3BHK0JXf4FbX7sStX+LSPzFtZ2FbDwQO59zLuTlhLLgB1/12VlbX1jc2c1v57Z3dvf3CwWHDqERTVqdKKN0KiWGCR6wOHARrxZoRGQrWDIc3E7/5xLThKnqEUcwCSfoR73FKwEp+6a4NXDKD7886haJbdqfAy8TLSBFlqHUKP+2uoolkEVBBjPE9N4YgJRo4FWycbyeGxYQOSZ/5lkbE5gTp9OQxPrVKF/eUti8CPFX/bqREGjOSoZ2UBAZm0ZuI/3l+Ar2rIOVRnACL6CyolwgMCk/+j7tcMwpiZAmhmttbMR0QTSjYluZSQjm2nXiLDSyTRqXsnZcrDxfF6nXWTg4doxNUQh66RFV0i2qojihS6AW9ojfn2Xl3PpzP2eiKk+0coTk4X7/bVpcF</latexit>

Matrix multiplication<latexit sha1_base64="DNGegz3UpEy/SoVyc/0rjZStnVY=">AAAB/3icbVDLSgMxFL1TX7W+qi7dBIvgqsxUQZdFNy4r2Ie0Q0nSTBuazAxJRihDF36DW127E7d+ikv/xEw7C9t6IHA4517uySGx4Nq47rdTWFvf2Nwqbpd2dvf2D8qHRy0dJYqyJo1EpDoEayZ4yJqGG8E6sWJYEsHaZHyb+e0npjSPwgcziZkv8TDkAafYWOmxJ7EZkQCRfrniVt0Z0CrxclKBHI1++ac3iGgiWWiowFp3PTc2foqV4VSwaamXaBZjOsZD1rU0xJJpP50FnqIzqwxQECn7QoNm6t+NFEutJ5LYySygXvYy8T+vm5jg2k95GCeGhXR+KEgEMhHKfo8GXDFqxMQSTBW3WREdYYWpsR0tXCFyajvxlhtYJa1a1buo1u4vK/WbvJ0inMApnIMHV1CHO2hAEyhIeIFXeHOenXfnw/mcjxacfOcYFuB8/QI84Ja+</latexit>

J<latexit sha1_base64="iPKhNFSPiDW86sRcCxgyDM+uv8o=">AAAB93icbVA9SwNBEN2LXzF+RS1tFoNgFe6ioGXQRqwSMB+QHGFvM5cs2d07dveE48gvsNXaTmz9OZb+EzfJFSbxwcDjvRlm5gUxZ9q47rdT2Njc2t4p7pb29g8Oj8rHJ20dJYpCi0Y8Ut2AaOBMQssww6EbKyAi4NAJJvczv/MMSrNIPpk0Bl+QkWQho8RYqfk4KFfcqjsHXideTiooR2NQ/ukPI5oIkIZyonXPc2PjZ0QZRjlMS/1EQ0zohIygZ6kkArSfzQ+d4gurDHEYKVvS4Ln6dyIjQutUBLZTEDPWq95M/M/rJSa89TMm48SApItFYcKxifDsazxkCqjhqSWEKmZvxXRMFKHGZrO0JRBTm4m3msA6adeq3lW11ryu1O/ydIroDJ2jS+ShG1RHD6iBWogiQC/oFb05qfPufDifi9aCk8+coiU4X79ELZNo</latexit>

X ⇥1 a ⇥ 2 b ⇥ 3 c<latexit sha1_base64="Mdj9E+YBNnyEN8zwAEnvpiuL0nQ=">AAACN3icbZDLSsNAFIYn9VbrLerSzWARXJWkFXRn0Y3LCvYCbQiT6aQdOpOEmYlQQh7Hh/AZ3OrClbgRt76BkzQF23pg4Oc75/Cf+b2IUaks690ora1vbG6Vtys7u3v7B+bhUUeGscCkjUMWip6HJGE0IG1FFSO9SBDEPUa63uQ263cfiZA0DB7UNCIOR6OA+hQjpZFrXg84UmOMWNJLB4pyIl07R54PUQHqcE68gjTmALtm1apZecFVYReiCopquebnYBjimJNAYYak7NtWpJwECUUxI2llEEsSITxBI9LXMkDaz0nyj6bwTJMh9EOhX6BgTv9uJIhLOeWenswOlMu9DP7X68fKv3ISGkSxIgGeGfkxgyqEWWpwSAXBik21QFhQfSvEYyQQVjrbBRePpzoTezmBVdGp1+xGrX5/UW3eFOmUwQk4BefABpegCe5AC7QBBk/gBbyCN+PZ+DC+jO/ZaMkodo7BQhk/v61NrX4=</latexit><latexit sha1_base64="4tiu7aGizBnvfz6YaX5e2UsfHTU=">AAACE3icbVDLSgMxFM3UV62vUZfdBIvgqsxUQZdFNy4r2Ae0Q0nSTBuaZIYkI5ShCz/Cb3Cra3fi1g9w6Z+YaUewrQcCh3Pu5dwcHHOmjed9OYW19Y3NreJ2aWd3b//APTxq6ShRhDZJxCPVwUhTziRtGmY47cSKIoE5bePxTea3H6jSLJL3ZhLTQKChZCEjyFip75Z7ApkRDiGCvcgwQTX8VXDfrXhVbwa4SvycVECORt/97g0ikggqDeFI667vxSZIkTKMcDot9RJNY0TGaEi7lkpk44J09okpPLXKAIaRsk8aOFP/bqRIaD0R2E5mB+plLxP/87qJCa+ClMk4MVSSeVCYcGgimDUCB0xRYvjEEkQUs7dCMkIKEWN7W0jBYmo78ZcbWCWtWtU/r9buLir167ydIiiDE3AGfHAJ6uAWNEATEPAInsELeHWenDfn3fmYjxacfOcYLMD5/AH7/Z4f</latexit>(I ⇥ J)

K

I

Linear transforma⌦bAB<latexit sha1_base64="PPCkjOkrnSVQ1Z4QUirmGsykYQg=">AAACC3icbVDLSsNAFL3xWesr2qWbwSK4KkkVdFnrxmUF+4A2lMl00g6dScLMRAiln+A3uNW1O3HrR7j0T5y0EWzrgQuHc+7lHo4fc6a043xZa+sbm1vbhZ3i7t7+waF9dNxSUSIJbZKIR7LjY0U5C2lTM81pJ5YUC5/Ttj++zfz2I5WKReGDTmPqCTwMWcAI1kbq26WewHrkB+gG/bJ63y47FWcGtErcnJQhR6Nvf/cGEUkEDTXhWKmu68Tam2CpGeF0WuwlisaYjPGQdg0NsaDKm8zCT9GZUQYoiKSZUKOZ+vdigoVSqfDNZhZQLXuZ+J/XTXRw7U1YGCeahmT+KEg40hHKmkADJinRPDUEE8lMVkRGWGKiTV8LX3wxNZ24yw2skla14l5UqveX5Vo9b6cAJ3AK5+DCFdTgDhrQBAIpPMMLvFpP1pv1bn3MV9es/KYEC7A+fwBbzZqC</latexit><latexit sha1_base64="CdDEFOmD07BOoEI/ujiP3GVwNE8=">AAACAXicbVDLSgMxFL1TX7W+qi7dBItQN2WmCrosutFdBfuA6VAyaaYNTTJDkhFK6cpvcKtrd+LWL3Hpn5i2s7CtBwKHc+7l3Jww4Uwb1/12cmvrG5tb+e3Czu7e/kHx8Kip41QR2iAxj1U7xJpyJmnDMMNpO1EUi5DTVji8nfqtJ6o0i+WjGSU0ELgvWcQINlbyy/cdwwTVyDvvFktuxZ0BrRIvIyXIUO8Wfzq9mKSCSkM41tr33MQEY6wMI5xOCp1U0wSTIe5T31KJbU4wnp08QWdW6aEoVvZJg2bq340xFlqPRGgnBTYDvexNxf88PzXRdTBmMkkNlWQeFKUcmRhN/496TFFi+MgSTBSztyIywAoTY1taSAnFxHbiLTewSprVindRqT5clmo3WTt5OIFTKIMHV1CDO6hDAwjE8AKv8OY8O+/Oh/M5H8052c4xLMD5+gWyTpbr</latexit><latexit sha1_base64="DNGegz3UpEy/SoVyc/0rjZStnVY=">AAAB/3icbVDLSgMxFL1TX7W+qi7dBIvgqsxUQZdFNy4r2Ie0Q0nSTBuazAxJRihDF36DW127E7d+ikv/xEw7C9t6IHA4517uySGx4Nq47rdTWFvf2Nwqbpd2dvf2D8qHRy0dJYqyJo1EpDoEayZ4yJqGG8E6sWJYEsHaZHyb+e0npjSPwgcziZkv8TDkAafYWOmxJ7EZkQCRfrniVt0Z0CrxclKBHI1++ac3iGgiWWiowFp3PTc2foqV4VSwaamXaBZjOsZD1rU0xJJpP50FnqIzqwxQECn7QoNm6t+NFEutJ5LYySygXvYy8T+vm5jg2k95GCeGhXR+KEgEMhHKfo8GXDFqxMQSTBW3WREdYYWpsR0tXCFyajvxlhtYJa1a1buo1u4vK/WbvJ0inMApnIMHV1CHO2hAEyhIeIFXeHOenXfnw/mcjxacfOcYFuB8/QI84Ja+</latexit><latexit sha1_base64="JtahwqYLiJdwEJa47CBn25f/Th8=">AAACAHicbVDLSgMxFL1TX7W+qi7dBIvgqsxUQZdFNy4r2Ae2Q8mkmTY0yQxJRihDN36DW127E7f+iUv/xEw7C9t6IHA4517uyQlizrRx3W+nsLa+sblV3C7t7O7tH5QPj1o6ShShTRLxSHUCrClnkjYNM5x2YkWxCDhtB+PbzG8/UaVZJB/MJKa+wEPJQkawsdJjT2AzCkKEUb9ccavuDGiVeDmpQI5Gv/zTG0QkEVQawrHWXc+NjZ9iZRjhdFrqJZrGmIzxkHYtlVhQ7aezxFN0ZpUBCiNlnzRopv7dSLHQeiICO5kl1MteJv7ndRMTXvspk3FiqCTzQ2HCkYlQ9n00YIoSwyeWYKKYzYrICCtMjC1p4UogprYTb7mBVdKqVb2Lau3+slK/ydspwgmcwjl4cAV1uIMGNIGAhBd4hTfn2Xl3PpzP+WjByXeOYQHO1y+V/5bn</latexit>

J<latexit sha1_base64="iPKhNFSPiDW86sRcCxgyDM+uv8o=">AAAB93icbVA9SwNBEN2LXzF+RS1tFoNgFe6ioGXQRqwSMB+QHGFvM5cs2d07dveE48gvsNXaTmz9OZb+EzfJFSbxwcDjvRlm5gUxZ9q47rdT2Njc2t4p7pb29g8Oj8rHJ20dJYpCi0Y8Ut2AaOBMQssww6EbKyAi4NAJJvczv/MMSrNIPpk0Bl+QkWQho8RYqfk4KFfcqjsHXideTiooR2NQ/ukPI5oIkIZyonXPc2PjZ0QZRjlMS/1EQ0zohIygZ6kkArSfzQ+d4gurDHEYKVvS4Ln6dyIjQutUBLZTEDPWq95M/M/rJSa89TMm48SApItFYcKxifDsazxkCqjhqSWEKmZvxXRMFKHGZrO0JRBTm4m3msA6adeq3lW11ryu1O/ydIroDJ2jS+ShG1RHD6iBWogiQC/oFb05qfPufDifi9aCk8+coiU4X79ELZNo</latexit>(I ⇥ 1)I<latexit sha1_base64="EZEmgoPYGnSmLv9tAdgJRCcw6xM=">AAACAXicbVDLSgMxFM34rPVVdekmWIS6KTNV0GXRjcsK9gHToWTSTBuaTIbkjlCGrvwGt7p2J279Epf+iWk7C9t6IHA4517OzQkTwQ247reztr6xubVd2Cnu7u0fHJaOjltGpZqyJlVC6U5IDBM8Zk3gIFgn0YzIULB2OLqb+u0npg1X8SOMExZIMoh5xCkBK/kVrwtcMoO9i16p7FbdGfAq8XJSRjkavdJPt69oKlkMVBBjfM9NIMiIBk4FmxS7qWEJoSMyYL6lMbE5QTY7eYLPrdLHkdL2xYBn6t+NjEhjxjK0k5LA0Cx7U/E/z08hugkyHicpsJjOg6JUYFB4+n/c55pREGNLCNXc3orpkGhCwba0kBLKie3EW25glbRqVe+yWnu4Ktdv83YK6BSdoQry0DWqo3vUQE1EkUIv6BW9Oc/Ou/PhfM5H15x85wQtwPn6BYuultM=</latexit>

Inner product<latexit sha1_base64="futOwDfv0EATwUk9NtBVM1D4uvk=">AAAB/3icbVDLSgMxFL3js9ZX1aWbYBFclZkq6LLUjcsK9iHtUDJppg1NMkOSEcrQhd/gVtfuxK2f4tI/MdPOwrYeCBzOuZd7coKYM21c99tZW9/Y3Nou7BR39/YPDktHxy0dJYrQJol4pDoB1pQzSZuGGU47saJYBJy2g/Ft5refqNIskg9mElNf4KFkISPYWOmxJ7AZBSGq90tlt+LOgFaJl5My5Gj0Sz+9QUQSQaUhHGvd9dzY+ClWhhFOp8VeommMyRgPaddSiQXVfjoLPEXnVhmgMFL2SYNm6t+NFAutJyKwk1lAvexl4n9eNzHhjZ8yGSeGSjI/FCYcmQhlv0cDpigxfGIJJorZrIiMsMLE2I4WrgRiajvxlhtYJa1qxbusVO+vyrV63k4BTuEMLsCDa6jBHTSgCQQEvMArvDnPzrvz4XzOR9ecfOcEFuB8/QIKgJae</latexit><latexit sha1_base64="ZrFty2l/Zyqi7tkI+zQEBL5il24=">AAACC3icbVDLSgMxFM3UV62v0S7dBIvgqsxUQZdVNy4r2Ae0Q0nSTBuaZIYkIwyln+A3uNW1O3HrR7j0T8y0I9jWAxcO59zLPRwcc6aN5305hbX1jc2t4nZpZ3dv/8A9PGrpKFGENknEI9XBSFPOJG0aZjjtxIoigTlt4/Ft5rcfqdIskg8mjWkg0FCykBFkrNR3yz2BzAiH8Br+Mtx3K17VmwGuEj8nFZCj0Xe/e4OIJIJKQzjSuut7sQkmSBlGOJ2WeommMSJjNKRdSyUSVAeTWfgpPLXKAIaRsiMNnKl/LyZIaJ0KbDezgHrZy8T/vG5iwqtgwmScGCrJ/FGYcGgimDUBB0xRYnhqCSKK2ayQjJBCxNi+Fr5gMbWd+MsNrJJWreqfV2v3F5X6Td5OERyDE3AGfHAJ6uAONEATEJCCZ/ACXp0n5815dz7mqwUnvymDBTifP44tmqI=</latexit><latexit sha1_base64="ZTPE9JepZqhbQcCQ/sGsbjQq6lY=">AAACDnicbVDLSgMxFM3UV62vUcGNm2ARXJWZKuiy6MZlhb6gHUuSZtrQJDMkGaGM/Qe/wa2u3Ylbf8Glf2LajmBbDwROzrmXezg45kwbz/tyciura+sb+c3C1vbO7p67f9DQUaIIrZOIR6qFkaacSVo3zHDaihVFAnPaxMObid98oEqzSNbMKKaBQH3JQkaQsVLXPeoIZAY4hAje1+DvB3fdolfypoDLxM9IEWSodt3vTi8iiaDSEI60bvtebIIUKcMIp+NCJ9E0RmSI+rRtqUSC6iCd5h/DU6v0YBgp+6SBU/XvRoqE1iOB7eQkoF70JuJ/Xjsx4VWQMhknhkoyOxQmHJoITsqAPaYoMXxkCSKK2ayQDJBCxNjK5q5gMbad+IsNLJNGueSfl8p3F8XKddZOHhyDE3AGfHAJKuAWVEEdEPAInsELeHWenDfn3fmYjeacbOcQzMH5/AGO6Juy</latexit>bB<latexit sha1_base64="94tL1q9O9cS77J172E010NqOwHQ=">AAAB/3icbVC7TsMwFL3hWcqrwMhiUSExVUlBgrHAwlgk+kBtVDmu01q1nch2kKqoA9/ACjMbYuVTGPkTnDYDbTmSpaNz7tU9PkHMmTau++2srK6tb2wWtorbO7t7+6WDw6aOEkVog0Q8Uu0Aa8qZpA3DDKftWFEsAk5bweg281tPVGkWyQczjqkv8ECykBFsrPTYFdgMgxBd90plt+JOgZaJl5My5Kj3Sj/dfkQSQaUhHGvd8dzY+ClWhhFOJ8VuommMyQgPaMdSiQXVfjoNPEGnVumjMFL2SYOm6t+NFAutxyKwk1lAvehl4n9eJzHhlZ8yGSeGSjI7FCYcmQhlv0d9pigxfGwJJorZrIgMscLE2I7mrgRiYjvxFhtYJs1qxTuvVO8vyrWbvJ0CHMMJnIEHl1CDO6hDAwgIeIFXeHOenXfnw/mcja44+c4RzMH5+gUI7Zad</latexit><latexit sha1_base64="DNGegz3UpEy/SoVyc/0rjZStnVY=">AAAB/3icbVDLSgMxFL1TX7W+qi7dBIvgqsxUQZdFNy4r2Ie0Q0nSTBuazAxJRihDF36DW127E7d+ikv/xEw7C9t6IHA4517uySGx4Nq47rdTWFvf2Nwqbpd2dvf2D8qHRy0dJYqyJo1EpDoEayZ4yJqGG8E6sWJYEsHaZHyb+e0npjSPwgcziZkv8TDkAafYWOmxJ7EZkQCRfrniVt0Z0CrxclKBHI1++ac3iGgiWWiowFp3PTc2foqV4VSwaamXaBZjOsZD1rU0xJJpP50FnqIzqwxQECn7QoNm6t+NFEutJ5LYySygXvYy8T+vm5jg2k95GCeGhXR+KEgEMhHKfo8GXDFqxMQSTBW3WREdYYWpsR0tXCFyajvxlhtYJa1a1buo1u4vK/WbvJ0inMApnIMHV1CHO2hAEyhIeIFXeHOenXfnw/mcjxacfOcYFuB8/QI84Ja+</latexit>

J<latexit sha1_base64="lARkly2hBl77atLgnGdkxY/CRV8=">AAAB/3icbVDLSgMxFL1TX7W+qi7dBIvgqsxUQZdFNy4r2Ie0Q8mkmTY0yQxJRihDF36DW127E7d+ikv/xEw7C9t6IHA4517uyQlizrRx3W+nsLa+sblV3C7t7O7tH5QPj1o6ShShTRLxSHUCrClnkjYNM5x2YkWxCDhtB+PbzG8/UaVZJB/MJKa+wEPJQkawsdJjT2AzCkKE++WKW3VnQKvEy0kFcjT65Z/eICKJoNIQjrXuem5s/BQrwwin01Iv0TTGZIyHtGupxIJqP50FnqIzqwxQGCn7pEEz9e9GioXWExHYySygXvYy8T+vm5jw2k+ZjBNDJZkfChOOTISy36MBU5QYPrEEE8VsVkRGWGFibEcLVwIxtZ14yw2sklat6l1Ua/eXlfpN3k4RTuAUzsGDK6jDHTSgCQQEvMArvDnPzrvz4XzORwtOvnMMC3C+fgE7TZa9</latexit>

I<latexit sha1_base64="6MgDcebxBw4AHPTGRkLXsNY4hn8=">AAAB93icbVA9SwNBEN2LXzF+RS1tFoNgFe6ioGXQRrsEzAckR9jbzCVLdveO3T3hOPILbLW2E1t/jqX/xE1yhUl8MPB4b4aZeUHMmTau++0UNja3tneKu6W9/YPDo/LxSVtHiaLQohGPVDcgGjiT0DLMcOjGCogIOHSCyf3M7zyD0iySTyaNwRdkJFnIKDFWaj4OyhW36s6B14mXkwrK0RiUf/rDiCYCpKGcaN3z3Nj4GVGGUQ7TUj/REBM6ISPoWSqJAO1n80On+MIqQxxGypY0eK7+nciI0DoVge0UxIz1qjcT//N6iQlv/YzJODEg6WJRmHBsIjz7Gg+ZAmp4agmhitlbMR0TRaix2SxtCcTUZuKtJrBO2rWqd1WtNa8r9bs8nSI6Q+foEnnoBtXRA2qgFqII0At6RW9O6rw7H87norXg5DOnaAnO1y9CmpNn</latexit>a(1 ⇥ 1 ⇥ 1)<latexit sha1_base64="xPJ872JvyatRpZQAUPNv3i1Wxnc=">AAACC3icbVDLSsNAFJ3UV62vaJduBotQNyWpgi6LblxWsA9oQ5lMJ+3QmSTM3Agh9BP8Bre6didu/QiX/onTNoi2HhjmcM69nMvxY8E1OM6nVVhb39jcKm6Xdnb39g/sw6O2jhJFWYtGIlJdn2gmeMhawEGwbqwYkb5gHX9yM/M7D0xpHoX3kMbMk2QU8oBTAkYa2OWq2wcumcY//9nArjg1Zw68StycVFCO5sD+6g8jmkgWAhVE657rxOBlRAGngk1L/USzmNAJGbGeoSExOV42P36KT40yxEGkzAsBz9XfGxmRWqfSN5OSwFgvezPxP6+XQHDlZTyME2AhXQQFicAQ4VkTeMgVoyBSQwhV3NyK6ZgoQsH09SfFl1PTibvcwCpp12vuea1+d1FpXOftFNExOkFV5KJL1EC3qIlaiKIUPaFn9GI9Wq/Wm/W+GC1Y+U4Z/YH18Q3BmJoj</latexit><latexit sha1_base64="lARkly2hBl77atLgnGdkxY/CRV8=">AAAB/3icbVDLSgMxFL1TX7W+qi7dBIvgqsxUQZdFNy4r2Ie0Q8mkmTY0yQxJRihDF36DW127E7d+ikv/xEw7C9t6IHA4517uyQlizrRx3W+nsLa+sblV3C7t7O7tH5QPj1o6ShShTRLxSHUCrClnkjYNM5x2YkWxCDhtB+PbzG8/UaVZJB/MJKa+wEPJQkawsdJjT2AzCkKE++WKW3VnQKvEy0kFcjT65Z/eICKJoNIQjrXuem5s/BQrwwin01Iv0TTGZIyHtGupxIJqP50FnqIzqwxQGCn7pEEz9e9GioXWExHYySygXvYy8T+vm5jw2k+ZjBNDJZkfChOOTISy36MBU5QYPrEEE8VsVkRGWGFibEcLVwIxtZ14yw2sklat6l1Ua/eXlfpN3k4RTuAUzsGDK6jDHTSgCQQEvMArvDnPzrvz4XzORwtOvnMMC3C+fgE7TZa9</latexit>

Node:=Tensor Edge:=Contraction Graph:=Tensor Network‣ The essence of the tensor network is multivariate polynomial.
‣ Tensor network decomposition = Polynomial Approximation

9

Various Tensor Network Models[Oseledets, 2011][Zhao et al, 2016][Tucker, 1966]‣ Given a task, which model is the best one?

① How to select the best model in an affordable manner?
② Is it possible to get models going beyond existing ones?

Part 2:
Structure Learning of TND

GraphicalRepresentationof TN
TN Structures
Structures
ReforgetheDiagram
Notation
Graphical
RepresentationofGraphical Representation of TN Structures
To build
build aa bijection
bijection fromfrom anan arbitrary
arbitrary simple
simple graph
graph totheTN
structure.
To
to
the
TN
structure.
To
To build a bijection from an arbitrary simple graph to the TN structure.
Graph
Graph
Graph
vertices
vertices
vertices

Tensor
network
Tensor
network
Tensor
network
cores/core
tensors/factors
cores/core
tensors/factors
cores/core
tensors/factors(a)product(a) matrix
matrixproduct(a) matrix
product

edges
contracted
indices
edges
contracted
indices
edges
contracted
indicesdegreenumberofindegreeof vertex
vertex number
numberof indicesindicesin eacheach
core
degreeof ofvertexof indicesin eachcorecoreweightweightof edgeedge
weight
of of
edge

upper
limit
of
upper
limit
of summation
summation
upper
limit
of summation

numbernumberofcontracted
numberof edgesedges number
numberof indicesindices
contracted
numberof ofedges
of indicescontracted(c) tensor
ring (TR)(c) tensor
ring (TR)(b)
train(b) tensor
tensortrain (TT)(TT)(b) tensor
train (TT)Correspondence
betweenGraphsstructures& Lim,
2019].
Correspondence
Correspondence
betweenGraphsandandTNTN
structures[Ye [Ye& Lim,
2019].

Additional rules

Weight-one edges are not allowed.
The isolated sub-graphs are “connected” by tensor outer product.
The vertices can be internal.

The TN structures can be fully described by its adjacency matrices.
Chao(RIKEN-AIP)
Chao
Li Li
(RIKEN-AIP)
Chao
Chao LiLi (RIKEN-AIP)(RIKEN-AIP)

chao.li@riken.jp
chao.li@riken.jp
chao.li@riken.jp
chao.li@riken.jp

June 208 / 18 8 / 18 12June 20June
8 / 18June 20

Unified Model Formulation‣ (Augmented) adjacency matrix: a simple way to model tensor networks
G1<latexit sha1_base64="pYi4WJk7jSecnWGY1T+9Z/pCQ/Y=">AAACA3icbVC7SgNBFL3rM8ZX1NJmMAhWYTcKWgYttIxgHpAsYXYymwyZmV1nZoWwpPQbbLW2E1s/xNI/cTbZwiQeuHA4517u4QQxZ9q47rezsrq2vrFZ2Cpu7+zu7ZcODps6ShShDRLxSLUDrClnkjYMM5y2Y0WxCDhtBaObzG89UaVZJB/MOKa+wAPJQkawsZLfFdgMCebp7aTn9Uplt+JOgZaJl5My5Kj3Sj/dfkQSQaUhHGvd8dzY+ClWhhFOJ8VuommMyQgPaMdSiQXVfjoNPUGnVumjMFJ2pEFT9e9FioXWYxHYzSykXvQy8T+vk5jwyk+ZjBNDJZk9ChOOTISyBlCfKUoMH1uCiWI2KyJDrDAxtqe5L4GY2E68xQaWSbNa8c4r1fuLcu06b6cAx3ACZ+DBJdTgDurQAAKP8AKv8OY8O+/Oh/M5W11x8psjmIPz9QuSS5ib</latexit>

G1

G2<latexit sha1_base64="vr70X2ljgj5Ik+S62HsF7i7pHcI=">AAACA3icbVC7SgNBFL3rM8ZX1NJmMAhWYTcKWgYttIxgHpAsYXYySYbMzK4zs0JYtvQbbLW2E1s/xNI/cTbZwiQeuHA4517u4QQRZ9q47rezsrq2vrFZ2Cpu7+zu7ZcODps6jBWhDRLyULUDrClnkjYMM5y2I0WxCDhtBeObzG89UaVZKB/MJKK+wEPJBoxgYyW/K7AZEcyT27RX7ZXKbsWdAi0TLydlyFHvlX66/ZDEgkpDONa647mR8ROsDCOcpsVurGmEyRgPacdSiQXVfjINnaJTq/TRIFR2pEFT9e9FgoXWExHYzSykXvQy8T+vE5vBlZ8wGcWGSjJ7NIg5MiHKGkB9pigxfGIJJorZrIiMsMLE2J7mvgQitZ14iw0sk2a14p1XqvcX5dp13k4BjuEEzsCDS6jBHdShAQQe4QVe4c15dt6dD+dztrri5DdHMAfn6xeT3pic</latexit><latexit sha1_base64="pYi4WJk7jSecnWGY1T+9Z/pCQ/Y=">AAACA3icbVC7SgNBFL3rM8ZX1NJmMAhWYTcKWgYttIxgHpAsYXYymwyZmV1nZoWwpPQbbLW2E1s/xNI/cTbZwiQeuHA4517u4QQxZ9q47rezsrq2vrFZ2Cpu7+zu7ZcODps6ShShDRLxSLUDrClnkjYMM5y2Y0WxCDhtBaObzG89UaVZJB/MOKa+wAPJQkawsZLfFdgMCebp7aTn9Uplt+JOgZaJl5My5Kj3Sj/dfkQSQaUhHGvd8dzY+ClWhhFOJ8VuommMyQgPaMdSiQXVfjoNPUGnVumjMFJ2pEFT9e9FioXWYxHYzSykXvQy8T+vk5jwyk+ZjBNDJZk9ChOOTISyBlCfKUoMH1uCiWI2KyJDrDAxtqe5L4GY2E68xQaWSbNa8c4r1fuLcu06b6cAx3ACZ+DBJdTgDurQAAKP8AKv8OY8O+/Oh/M5W11x8psjmIPz9QuSS5ib</latexit>

G2<latexit sha1_base64="vr70X2ljgj5Ik+S62HsF7i7pHcI=">AAACA3icbVC7SgNBFL3rM8ZX1NJmMAhWYTcKWgYttIxgHpAsYXYySYbMzK4zs0JYtvQbbLW2E1s/xNI/cTbZwiQeuHA4517u4QQRZ9q47rezsrq2vrFZ2Cpu7+zu7ZcODps6jBWhDRLyULUDrClnkjYMM5y2I0WxCDhtBeObzG89UaVZKB/MJKK+wEPJBoxgYyW/K7AZEcyT27RX7ZXKbsWdAi0TLydlyFHvlX66/ZDEgkpDONa647mR8ROsDCOcpsVurGmEyRgPacdSiQXVfjINnaJTq/TRIFR2pEFT9e9FgoXWExHYzSykXvQy8T+vE5vBlZ8wGcWGSjJ7NIg5MiHKGkB9pigxfGIJJorZrIiMsMLE2J7mvgQitZ14iw0sk2a14p1XqvcX5dp13k4BjuEEzsCDS6jBHdShAQQe4QVe4c15dt6dD+dztrri5DdHMAfn6xeT3pic</latexit>

G2 I2<latexit sha1_base64="vr70X2ljgj5Ik+S62HsF7i7pHcI=">AAACA3icbVC7SgNBFL3rM8ZX1NJmMAhWYTcKWgYttIxgHpAsYXYySYbMzK4zs0JYtvQbbLW2E1s/xNI/cTbZwiQeuHA4517u4QQRZ9q47rezsrq2vrFZ2Cpu7+zu7ZcODps6jBWhDRLyULUDrClnkjYMM5y2I0WxCDhtBeObzG89UaVZKB/MJKK+wEPJBoxgYyW/K7AZEcyT27RX7ZXKbsWdAi0TLydlyFHvlX66/ZDEgkpDONa647mR8ROsDCOcpsVurGmEyRgPacdSiQXVfjINnaJTq/TRIFR2pEFT9e9FgoXWExHYzSykXvQy8T+vE5vBlZ8wGcWGSjJ7NIg5MiHKGkB9pigxfGIJJorZrIiMsMLE2J7mvgQitZ14iw0sk2a14p1XqvcX5dp13k4BjuEEzsCDS6jBHdShAQQe4QVe4c15dt6dD+dztrri5DdHMAfn6xeT3pic</latexit><latexit sha1_base64="Eo0ma+UOe+TgbgwvBOtGvsWH0oA=">AAAB+XicbVA9SwNBEJ2LXzF+RS1tFoNgFe6ioGXQRruI5gOSI+xt9pIlu3vH7p4QjvwEW63txNZfY+k/cS+5wiQ+GHi8N8PMvCDmTBvX/XYKa+sbm1vF7dLO7t7+QfnwqKWjRBHaJBGPVCfAmnImadMww2knVhSLgNN2ML7N/PYzVZpF8slMYuoLPJQsZAQbKz3e92v9csWtujOgVeLlpAI5Gv3yT28QkURQaQjHWnc9NzZ+ipVhhNNpqZdoGmMyxkPatVRiQbWfzk6dojOrDFAYKVvSoJn6dyLFQuuJCGynwGakl71M/M/rJia89lMm48RQSeaLwoQjE6HsbzRgihLDJ5Zgopi9FZERVpgYm87ClkBMbSbecgKrpFWrehfV2sNlpX6Tp1OEEziFc/DgCupwBw1oAoEhvMArvDmp8+58OJ/z1oKTzxzDApyvX3HFlAw=</latexit>

G1

Edges

<latexit sha1_base64="pYi4WJk7jSecnWGY1T+9Z/pCQ/Y=">AAACA3icbVC7SgNBFL3rM8ZX1NJmMAhWYTcKWgYttIxgHpAsYXYymwyZmV1nZoWwpPQbbLW2E1s/xNI/cTbZwiQeuHA4517u4QQxZ9q47rezsrq2vrFZ2Cpu7+zu7ZcODps6ShShDRLxSLUDrClnkjYMM5y2Y0WxCDhtBaObzG89UaVZJB/MOKa+wAPJQkawsZLfFdgMCebp7aTn9Uplt+JOgZaJl5My5Kj3Sj/dfkQSQaUhHGvd8dzY+ClWhhFOJ8VuommMyQgPaMdSiQXVfjoNPUGnVumjMFJ2pEFT9e9FioXWYxHYzSykXvQy8T+vk5jwyk+ZjBNDJZk9ChOOTISyBlCfKUoMH1uCiWI2KyJDrDAxtqe5L4GY2E68xQaWSbNa8c4r1fuLcu06b6cAx3ACZ+DBJdTgDurQAAKP8AKv8OY8O+/Oh/M5W11x8psjmIPz9QuSS5ib</latexit>

I1<latexit sha1_base64="O/m4HxTLUmr6XSW2cPgdyWfZvck=">AAAB+XicbVA9SwNBEJ2LXzF+RS1tFoNgFe6ioGXQRruI5gOSI+xt9pIlu3vH7p4QjvwEW63txNZfY+k/cS+5wiQ+GHi8N8PMvCDmTBvX/XYKa+sbm1vF7dLO7t7+QfnwqKWjRBHaJBGPVCfAmnImadMww2knVhSLgNN2ML7N/PYzVZpF8slMYuoLPJQsZAQbKz3e971+ueJW3RnQKvFyUoEcjX75pzeISCKoNIRjrbueGxs/xcowwum01Es0jTEZ4yHtWiqxoNpPZ6dO0ZlVBiiMlC1p0Ez9O5FiofVEBLZTYDPSy14m/ud1ExNe+ymTcWKoJPNFYcKRiVD2NxowRYnhE0swUczeisgIK0yMTWdhSyCmNhNvOYFV0qpVvYtq7eGyUr/J0ynCCZzCOXhwBXW4gwY0gcAQXuAV3pzUeXc+nM95a8HJZ45hAc7XL3AylAs=</latexit>

order-6 Tensor Ring

(Free legs) or zeros(Augmented) Adjacency matrix‣ Tensor networks (TN) are fully modeled by adjacency matrices.
‣ The matrix contains the both topological and algebraic information.
13

Problem
Formulation
UnifiedProblem FormulationModel Formulation
A general model of TN representation with arbitrary graphical structure.
A general model of TN representation with arbitrary graphical structure.
X = TN(V; A),
X = TN(V; A),

(1)
(1)

where TN( · ; A) denotes the index contraction operations under the adjawhere TN( · ; A) denotes
the index contraction operations under the adjaN◊N
cency matrix A œ R N◊N and V := {Vi , i œ [N]} denotes a collection ofcency matrix A œ R
and V := {Vi , i œ [N]} denotes a collection ofcore tensors.
core tensors.
Problem setting - structure learning of tensor network (TN) decomposition

Formal statement

Given a population of tensors {X }, the structure learning of TN is to findthe optimal adjacency matrix A0 , such that for every X(i) œ {X } there‚ that satisfies Eq. (1).
exists a set of tensors V(i)
Chao
Chao LiLi (RIKEN-AIP)(RIKEN-AIP)

chao.li@riken.jp
chao.li@riken.jp

June 20
20
June

14

18
99//18

Topology Search via Evolutionary Algorithm
Topology Search via Evolutionary Algorithm

Optimization Model

Optimization model

Topology

Optimization modelmin ÎD(A) · log(A)Î1 ,Approx. error

A

B

.
.2
.
.
‚
s.t. EX min
Æ”(2)
A .X ≠ TN(V; A).
B
F.2
.
AœA
‚
V
.
.
‚
Squarenessmin ÎD(A) · log(A)Î1 , s.t. EX min .X ≠ TN(V; A).
Æ”
(2)
F
AœA
‚
V
where D( · ) denotes filling the non-diagonal elements of a matrix by zeros,log( · ) and Î · Î denotes the element-wise logarithm and 1-norm of a matrix.
where D( · ) denotes filling the non-diagonal elements of a matrix by zeros,log( · ) and Î(roughly)· Î denotes the element-wise logarithm and 1-norm of a matrix.
Proposition
The
logarithm(roughly)of models’ compression ratio is bounded by the objective
Proposition
function in Eq. (2) up to a fixed affine transform.
The logarithm of models’ compression ratio is bounded by the objectivefunction in Eq. (2) up to a fixed affine transform.

Integer programming chao.li@riken.jp
is generally NP-complete. June 20

Chao Li (RIKEN-AIP)

Chao Li (RIKEN-AIP)

chao.li@riken.jp

June 20

10 / 18

10 / 18

15

Part 3:TN meets Genetic Algorithm

Genetic Algorithm: An Efficient Heuristic
34(5#*,64&7#*8

1*)2/>#?4$$'(.

Pros:
Global optimization@2&'0'A)&'#(

B#$%&'#(!"#$%&'#()*+,-$.#*'&/0

Cons:

BCDD@939-:;!6‣‣ Multiobjective friendly‣ Parallel computation;442,<4)*('(.
=*)047#*8

3=E$'F*)*+,G#*,H#02%&)&'#()$,34(5#*,64&7#*8,942*454(&)&'#(,(CTNR)‣ Prohibitive computation‣ No theoretical guarantee

The computational requirement is partially satisfied by the boost
development of GPUs (HPC-Tensorflow).
17

Structure Encoding
Encoding the Adjacency matrices into fixed length strings.
Adjacency Matrix§0 2 0 0·¨¸ Binarization¨2 0 2 0¸¨0 2 0 2¸¨¸
0 0 2 0¹©
Order-4 tensor with TT-format

Binary form

§0
¨
¨1
¨0
¨
©0

1 0 0 · Encoding Chromosomes (Population)
TT
¸
1
0
0
1
0
1
0 1 0¸
1 0 1 1 0 1 TR
1 0 1¸¸
Gene (Allele=0)
0 1 0¹

Adjacency Matrix
Chromosomes (Population)

2

0 1 3

0 4

2

0 5 3

0 4

Gene (Allele=0)18

Meta-algorithm

19Genetic Operators: Fitness and Elimination
Genetic Operators: Fitness and Elimination
Genetic
Operators:
Fitness and Elimination
1. Fitness
score:

1. Fitness score:
1. Fitness score:
.
.
.
.2
.
.
.
.
2
‚
‚
‚
‚
‚
.
.
.
.
F (A) =. .D(A) · log(A)
+
⁄
·
min
X
≠
TN(
V;
A)
/
ÎX
Î
,
(3)
.
.
.
2
F
.
.
.
2
‚ = .D(A)‚ · log(A)‚ . +‚ . / FÎX Î ,‚ A)
1 ⁄ · minV‚.X ≠ TN(V;
F (A)
(3)
F
1

¸

‚
V

¸

˚˙

F

˚˙
˝
relative squareerror (RSE)
relative square error (RSE)˝

The fitness scores are used to rank the individuals.
The fitness scores are used to rank the individuals.
2. Elimination

2. Elimination
high

C1
C2

low

C99
C100

!"##$
20

Parent
selection(Russianroulette
process)
Genetic
Operators:
RecombinationRecombination
Genetic
Operators:
3-1: Parent
0.3 selection (Russian roulette process)3.1 Parent selection (Russian roulette process)

;

0.3

Fixed
1.4Point

1.2
A–
Pr = max; 0.01, 3ln
4<– + — · rankeps
Pr = max 0.01, ln
,

4.2

Point

B

4.28.2
1.4
3.21.2AC

BD C

Crossover:
3.2 Crossover

Parents

8.2

3.2
E D FE

spin thespin theroulette
roulette
wheelwheel

eps + — · rank

4

where –, — > 0 denote tuning param
where –, — > 0 denote tuning parameters.
eters.

F

0 1 1 0 0 1

Recombination(crossover)1 1 0 1 0 0

Chao Li (RIKEN-AIP)

3

Children0 1 1 1 0 0
1 1 0 0 0 1

chao.li@riken.jp

June 20

13 / 18

21

Genetic Operators: Mutation
4. Mutation

1 1 0 0 0 1

Mutation

1 1 0 0 1 1

Discussion：‣ Mutation is the key operator to obtain the global minimum.
‣ The algorithm design follows the maximum entropy principle.
‣ More cost results in better performance.
22

Tensor Decomposition
on Synthetic
Data
Numerical
Experiments on Synthetic
Data
Numerical Experiments
Experiments on
Numerical
on Synthetic
SyntheticData
Data
The synthetic data are randomly (Gaussian, ‡ = 0.1) generated with ranThesynthetic
data areare randomly(Gaussian,
‡‡4,
==
with
ran-ranThedatarandomly(Gaussian,withdomsynthetic
graph (Erods-Renyi
model G(n,p), n =
.0.1)
. .0.1), 8,generated
pgenerated
œ [0.15,0.85]).
domgraphgraph(Erods-Renyi(Erods-Renyi modelœœ
[0.15,
0.85]).
dommodel G(n,G(n,p),p),nn==4,4,. ....,.8,, 8,p p[0.15,
0.85]).

Gen.¿ – The number of generations we obtain the results.
Gen.¿
The numberof generations
wetheobtainthe results.
Eff.ø –––TheParameter
ratio
compared to
ground-truth.
Gen.¿
numberof generations
we obtain
the results.
Eff.ø – Parameter ratio compared to the ground-truth.
Eff.ø – Parameter ratio compared to the ground-truth.
Order
Order
Order
4
5
6
7
8
Trial
4 Eff.
5 Eff.
6
7
8
Trial
Gen.
Gen.
Gen.
Eff.
Gen.
Eff.
Gen.
Eff.
4
5
6
7
8
Trial
Gen.
Eff.
Gen.
Eff.
Gen.
Eff.
Gen.
Eff.
Gen.
Eff.
Eff.
Eff.
Gen.
Eff.
Gen.
Eff.
A Gen.
001 Eff.
1.333 Gen.
003
1.084 Gen.
005
1.052
003
1.052
001
1.396
A
001
1.333
003
1.084
005
1.052
003
1.052
001
1.396
B
001
1.500
003
1.690
006
1.062
008
1.096
005
1.000
AB
001
1.333
003
1.084
005
1.052
003
1.052
001
1.396
001
1.500
003
1.690
006
1.062
008
1.096
005
1.000
C
001 1.500
1.500 003
002 1.690
1.000 006
004 1.062
1.000 008
003
1.000
001
1.320
BC
001
1.096
005
1.000
001
1.500
002
1.000
004
1.000
003
1.000
001
1.320
D
001
1.000
001
1.445
003
1.000
001
1.000
020
1.000
CD
001
002
1.000 003
004 1.000
1.000 001
003 1.000
1.000 020001 1.000
1.320
001 1.500
1.000
001
1.445
E
001
1.000
002
1.000
005
1.052
003
1.000
005
1.000
001 1.000
1.000
002
1.000
DE
001
001
1.445 005
003 1.052
1.000 003
001 1.000
1.000 005020 1.000
1.000
E
001
1.000
002
1.000
005
1.052
003
1.000
005
1.000
The proposed
proposed method
methodobtains
obtainsthe
thesamesame(Eff.=(Eff.=
1.000)
even
better(Eff.>
1.000)
comThe1.000)or or
even
better(Eff.>
1.000)
compression ratio
ratio compared
comparedtotothetheground-truth.
ground-truth.
pressionThe
proposed method
obtains thesame (Eff.= 1.000) or even better (Eff.> 1.000) compression ratio compared to the ground-truth.
Chao LiLi (RIKEN-AIP)(RIKEN-AIP)
Chaochao.li@riken.jp
chao.li@riken.jp

JuneJune
20 20 14 / 14
18 / 1823

Convergence8765# of individuals=25# of individuals=50# of individuals=100

1.5

0.55

1

0.5

8
7
6

# of individuals=25# of individuals=50# of individuals=100# of individuals=25# of individuals=50# of individuals=100

54Fitness

Fitness

Fitness# of individuals=25# of individuals=50# of individuals=100

5.5
5
4.5
4
3.5
3
2.5
2

Fitness

0.9
0.85
0.8
0.75
0.7
0.65
0.6

3

4
3

2
2

0.45
0

5

10

15

20

25

Number of generations

300(a) min

5

10

15

20

25

Number of generations

0

30

5

(b) median

10

15

20

25

Number of generations

30

0

5

(c) mean

10

15

20

25

Number of generations

30(d) std

Convergence with various number of individuals

Fitness

Fitness

0.7
0.65
0.6
0.55

5
4.5
4
3.5
3
2.5
2

8
7
6
5

Perc. of elimination=0.9
Perc. of elimination=0.8
Perc. of elimination=0.6
Perc. of elimination=0.2

1.5

8
7
6
5

Perc. of elimination=0.9
Perc. of elimination=0.8
Perc. of elimination=0.6
Perc. of elimination=0.2

4

Fitness

Perc. of elimination=0.9
Perc. of elimination=0.8
Perc. of elimination=0.6
Perc. of elimination=0.2

Fitness

0.85
0.8
0.75

3
2

1

Perc. of elimination=0.9
Perc. of elimination=0.8
Perc. of elimination=0.6
Perc. of elimination=0.2

4
3
2

0.5
0

5

10

15

20

25

Number of generations(a) min

30

0

5

10

15

20

25

Number of generations(b) median

30

0

5

10

15

20

25

Number of generations

300(c) mean

5

10

15

20

25

Number of generations

30(d) std

Convergence with various elimination percentage
Trade off between “elitism” and “populism”.
24Numerical
ImagesNumerical Experiments
Experiments on Natural Images

TN Decomposition on Natural Images

10
LIVE dataset
dataset [Sheikh[Sheikh etetal.,al.,
10 natural
natural images
images areare randomly
randomly selected from LIVE2006],
2006], andand reshaped
reshaped asas order-8order-8 tensors.

The learned
learned topology
topology have
have more
more complex
complex structures
The
structures than
than simple
simple ones
ones likelike
line,tree
tree oror cycles.
cycles.
line,
The learned structures are more complex than path, tree or cyclic graphs
Chao Li (RIKEN-AIP)
Chao Li (RIKEN-AIP)

chao.li@riken.jp
chao.li@riken.jp

June 20
June 20

16 / 18
16 / 1825

Discussion and Concluding Remarks
Computationally efficient? No, and yes.

‣ Additional computation requirement is unavoidable.
‣ Lots of near-optimal structures outperforms the existing models.
Potential applications? Model compression.
Takeaway messages and future work? Small-world network.

‣ The learned topology is far away from the well-developed models‣ Intuitively the results are more close to Watts–Strogatz graphs[Watts and Strogatz, Nature, 1998]

26