‭Seminar on Reservoir Computing and its Applications‬‭Date and Time‬‭: July 18 (Thu), 2024, 15:00-18:00‬‭Venue‬‭: Room 611B, 6th floor, Building 2, Nagoya Institute‬‭of Technology [‬‭MAP‬‭]‬‭Organizer‬‭: Gouhei Tanaka (Dept. Computer Science,‬‭NITech)‬‭Program‬‭15:00-15:50‬‭Denis Kleyko‬‭(Örebro University & Research Institutes of Sweden)‬‭Title‬‭: On Interplay Between Reservoir Computing & Hyperdimensional Computing‬‭ bstract‬‭:‬ ‭Both‬ ‭reservoir‬ ‭computing‬ ‭and‬ ‭hyperdimensional‬ ‭computing‬ ‭rely‬ ‭on‬ ‭the‬ ‭idea‬ ‭of‬A‭computing‬ ‭with‬‭randomized‬‭distributed‬‭representations.‬‭Over‬‭the‬‭years,‬‭we‬‭have‬‭seen‬‭how‬‭the‬‭two‬‭could‬‭interplay‬‭in‬‭various‬‭ways‬‭and‬‭I‬‭will‬‭present‬‭some‬‭of‬‭these‬‭observations‬‭in‬‭the‬‭talk.‬ ‭The‬ ‭talk‬ ‭will‬ ‭include‬ ‭three‬ ‭related‬ ‭narratives‬ ‭(if‬ ‭time‬ ‭allows).‬ ‭We‬ ‭will‬ ‭begin‬ ‭by‬ ‭first‬‭introducing‬ ‭how‬ ‭the‬ ‭ideas‬ ‭used‬ ‭for‬ ‭analyzing‬ ‭the‬ ‭memory‬ ‭capacity‬ ‭of‬ ‭distributed‬‭representations‬ ‭lead‬ ‭to‬ ‭understanding‬ ‭the‬ ‭mechanisms‬ ‭affecting‬ ‭the‬ ‭capacity‬ ‭of‬ ‭reservoir‬‭computing‬‭models.‬‭Second,‬‭the‬‭capacity‬‭analysis‬‭involves‬‭making‬‭parallels‬‭between‬‭the‬‭two‬‭frameworks,‬ ‭which‬ ‭in‬ ‭turn,‬ ‭allows‬ ‭leveraging‬ ‭the‬ ‭ideas‬ ‭from‬ ‭hyperdimensional‬ ‭computing‬‭models‬‭that‬‭use‬‭integer-valued‬‭representations‬‭(e.g.,‬‭multiply-add-permute‬‭model)‬‭to‬‭design‬‭minimalistic‬ ‭echo‬ ‭state‬ ‭networks‬ ‭amenable‬ ‭to‬ ‭realization‬ ‭in‬ ‭digital‬ ‭hardware‬ ‭(e.g.,‬ ‭field‬‭programmable‬ ‭gate‬ ‭arrays).‬‭Third,‬‭there‬‭is‬‭a‬‭recent‬‭proposal‬‭for‬‭reservoir‬‭computing‬‭(“next‬‭generation‬ ‭reservoir‬ ‭computing”‬ ‭)‬ ‭that‬ ‭suggests‬ ‭that‬ ‭the‬ ‭reservoir‬ ‭should‬ ‭be‬ ‭formed‬‭using‬‭localist‬ ‭representations‬ ‭as‬ ‭an‬ ‭explicit‬ ‭concatenation‬ ‭of‬ ‭the‬ ‭memory‬ ‭buffer‬ ‭with‬ ‭some‬‭time-delayed‬ ‭states‬ ‭and‬ ‭the‬ ‭corresponding‬ ‭higher-order‬ ‭interactions‬ ‭of‬ ‭the‬ ‭states‬ ‭in‬ ‭the‬‭buffer.‬ ‭We‬ ‭demonstrate‬ ‭that‬ ‭this‬ ‭proposal‬ ‭could‬ ‭be‬ ‭first‬ ‭interpreted‬‭as‬‭a‬‭polynomial‬‭kernel‬‭machine‬ ‭that‬ ‭in‬ ‭turn‬ ‭could‬ ‭be‬ ‭approximated‬ ‭with‬ ‭randomized‬ ‭compositional‬ ‭distributed‬‭representations‬ ‭using‬ ‭the‬ ‭ideas‬ ‭of‬ ‭self-binding‬ ‭from‬ ‭hyperdimensional‬ ‭computing.‬ ‭Our‬‭proposal:‬ ‭a)‬ ‭brings‬ ‭the‬ ‭idea‬ ‭of‬ ‭higher-order‬ ‭interactions‬ ‭into‬ ‭the‬ ‭design‬ ‭space‬‭of‬‭reservoir‬‭computing‬ ‭models‬ ‭operating‬ ‭with‬ ‭randomized‬ ‭representations;‬ ‭b)‬ ‭provides‬ ‭computational‬‭savings‬‭(in‬‭terms‬‭of‬‭the‬‭reservoir‬‭size);‬‭and‬‭c)‬‭has‬‭an‬‭elegant‬‭realization‬‭via‬‭a‬‭circuit‬‭that‬‭is‬‭based‬ ‭on‬ ‭Sigma-Pi‬ ‭neurons‬ ‭that‬ ‭can‬ ‭iteratively‬ ‭compute‬ ‭distributed‬ ‭representations‬ ‭of‬‭higher-order‬‭interactions‬‭for‬‭an‬‭arbitrary‬‭order.‬‭A‬‭proof-of-concept‬‭of‬‭the‬‭proposed‬‭approach‬‭has been implemented on a neuromorphic chip Loihi 2.‬‭16:00-16:50‬‭Ziqiang Li‬‭(The University of Tokyo)‬‭ itle‬‭: Multiple-Reservoir Echo State Network: A Promising Reservoir‬
T‭Computing-based Model for Processing Time Series Tasks in Edge Computing‬‭Scenarios‬‭ bstract‬‭:‬‭Edge‬‭computing‬‭is‬‭a‬‭new‬‭computing‬‭paradigm‬‭that‬‭aims‬‭to‬‭address‬‭the‬‭limitations‬A‭of‬ ‭traditional‬ ‭cloud‬ ‭computing‬ ‭in‬ ‭handling‬ ‭data‬ ‭generated‬ ‭by‬ ‭an‬ ‭increasing‬ ‭number‬ ‭of‬‭Internet‬ ‭of‬ ‭Things‬ ‭(IoT)‬ ‭devices‬ ‭connected‬ ‭to‬ ‭the‬ ‭Internet.‬ ‭The‬ ‭concept‬ ‭of‬ ‭Edge‬ ‭Machine‬‭ earning‬‭(Edge‬‭ML)‬‭strives‬‭to‬‭facilitate‬‭the‬‭training‬‭and‬‭deployment‬‭of‬‭ML‬‭models‬‭directly‬‭on‬L‭edge‬‭devices,‬‭encompassing‬‭both‬‭Edge‬‭Learning‬‭(EL)‬‭and‬‭Edge‬‭Inference‬‭(EI).‬‭At‬‭present,‬‭edge‬ ‭inference‬ ‭has‬ ‭been‬ ‭widely‬ ‭applied‬ ‭by‬ ‭means‬ ‭of‬ ‭developed‬ ‭edge‬ ‭devices.‬ ‭However,‬‭Edge‬ ‭Learning‬ ‭(EL,‬ ‭i.e.,‬ ‭training‬ ‭on‬ ‭edge‬ ‭devices)‬ ‭is‬ ‭still‬ ‭a‬ ‭promising‬ ‭paradigm‬ ‭since‬ ‭it‬‭requires‬ ‭local‬ ‭model‬ ‭training‬ ‭with‬ ‭constrained‬ ‭computational‬ ‭resources‬‭and‬‭limited‬‭training‬‭data.‬‭Reservoir‬‭Computing‬‭(RC)‬‭is‬‭a‬‭promising‬‭solution‬‭for‬‭dealing‬‭with‬‭time-series‬‭machine‬‭learning‬‭tasks‬‭on‬‭edge.‬‭In‬‭this‬‭talk,‬‭an‬‭RC-based‬‭framework‬‭named‬‭multiple-reservoir‬‭echo‬‭state‬ ‭network‬ ‭(MRESN)‬ ‭has‬ ‭become‬ ‭more‬ ‭and‬ ‭more‬ ‭popular‬ ‭because‬ ‭of‬ ‭its‬ ‭high‬‭effectiveness‬‭and‬‭efficiency.‬‭Based‬‭on‬‭this‬‭background,‬‭an‬‭emerging‬‭problem‬‭is‬‭to‬‭improve‬‭the‬ ‭computational‬ ‭ability‬ ‭of‬ ‭MRESN.‬ ‭I‬ ‭will‬ ‭present‬ ‭my‬ ‭recent‬ ‭work‬ ‭on‬ ‭optimizing‬ ‭MRESN‬‭from‬ ‭the‬ ‭perspective‬ ‭of‬ ‭searching‬ ‭for‬ ‭a‬ ‭suitable‬ ‭topology‬ ‭corresponding‬ ‭to‬ ‭a‬ ‭given‬‭time-series‬ ‭task.‬ ‭In‬ ‭this‬ ‭work,‬ ‭a‬ ‭unified‬ ‭approach‬ ‭is‬ ‭proposed‬ ‭to‬ ‭represent‬‭the‬‭topology‬‭of‬‭MRESN.‬ ‭Moreover,‬ ‭a‬ ‭genetic‬ ‭algorithm-based‬ ‭method‬‭is‬‭proposed‬‭to‬‭automatically‬‭search‬‭for‬ ‭an‬ ‭appropriate‬ ‭network‬ ‭topology.‬ ‭Lastly,‬ ‭the‬ ‭experimental‬ ‭results‬ ‭will‬ ‭indicate‬ ‭that‬‭topology is an important factor that affects computational ability.‬‭17:00-17:50‬‭Hiroto Tamura‬‭(The University of Tokyo)‬‭ itle‬‭: Temporal Information Processing Using Mahalanobis Distance of Reservoir‬
T‭States (MD-RS)‬‭Abstract‬‭:‬‭ eservoir‬‭computing‬‭is‬‭increasingly‬‭recognized‬‭as‬‭an‬‭efficient‬‭machine‬‭learning‬‭method‬‭for‬R‭time-series‬ ‭processing,‬ ‭notable‬ ‭for‬ ‭its‬ ‭low‬ ‭learning‬ ‭cost‬ ‭and‬ ‭reduced‬ ‭training‬ ‭data‬‭requirements.‬ ‭Departing‬ ‭from‬ ‭the‬ ‭conventional‬ ‭linear‬ ‭regression‬ ‭readout‬ ‭used‬ ‭in‬ ‭standard‬‭reservoir‬‭computing,‬‭this‬‭study‬‭introduces‬‭a‬‭"distance-based‬‭readout"‬‭method.‬‭This‬‭method‬‭is‬ ‭especially‬ ‭effective‬ ‭for‬ ‭tasks‬ ‭requiring‬ ‭differentiation,‬ ‭such‬ ‭as‬ ‭anomaly‬ ‭detection‬ ‭and‬‭classification.‬‭ irst,‬ ‭we‬ ‭introduce‬ ‭the‬ ‭Mahalanobis‬ ‭Distance‬ ‭of‬ ‭Reservoir‬ ‭States‬ ‭(MD-RS)‬ ‭for‬ ‭online‬
F‭time-series‬ ‭anomaly‬ ‭detection.‬ ‭In‬ ‭the‬ ‭MD-RS‬ ‭method,‬ ‭the‬ ‭reservoir's‬ ‭responses‬‭to‬‭normal‬‭time‬‭series‬‭are‬‭fitted‬‭to‬‭a‬‭multivariate‬‭Gaussian‬‭distribution.‬‭During‬‭testing,‬‭the‬‭Mahalanobis‬‭distance‬ ‭from‬ ‭this‬ ‭fitted‬ ‭distribution‬ ‭is‬ ‭used‬ ‭as‬ ‭a‬ ‭measure‬ ‭of‬ ‭anomaly.‬ ‭We‬ ‭conducted‬ ‭a‬‭comprehensive‬ ‭performance‬ ‭evaluation‬ ‭using‬ ‭the‬ ‭UCR‬ ‭Anomaly‬ ‭Archive‬ ‭to‬ ‭compare‬ ‭the‬‭state-of-the-art methods with our MD-RS approach.‬‭ urther,‬ ‭we‬ ‭extend‬ ‭MD-RS‬ ‭for‬ ‭time‬ ‭series‬ ‭classification.‬ ‭In‬ ‭this‬ ‭method,‬ ‭the‬ ‭reservoir's‬F‭responses‬‭to‬‭each‬‭class‬‭are‬‭fitted‬‭to‬‭a‬‭multivariate‬‭Gaussian‬‭distribution,‬‭and‬‭classification‬‭is‬‭based‬ ‭on‬ ‭the‬ ‭distance‬‭(measured‬‭as‬‭Kullback–Leibler‬‭divergence)‬‭between‬‭the‬‭distribution‬‭of‬‭each‬‭class‬‭and‬‭that‬‭of‬‭the‬‭response‬‭during‬‭testing.‬‭We‬‭evaluated‬‭the‬‭performance‬‭of‬‭the‬‭proposed‬ ‭readout‬ ‭and‬ ‭standard‬ ‭readouts‬ ‭using‬ ‭linear‬ ‭regression‬ ‭or‬ ‭support‬ ‭vector‬‭machines,‬ ‭using‬ ‭four‬ ‭time-series‬ ‭classification‬ ‭benchmarks.‬ ‭Our‬ ‭proposed‬ ‭method‬‭demonstrates superior performance compared to existing methods.‬