11th Workshop 2022 on Observatory & Measurement Technology from Visible to Infrared

5Outline
23th December 2022
10:20 – 10:55
11th Workshop 2022 on Observatory & Measurement Technology from Visible to Infrared@ Kyoto University

高谷 裕浩
Yasuhiro Takaya
Osaka University

NanoSmart@Takaya & Mizutani Lab.

2

3

4

Environment
Adaptive
Metrology;

PhotonMetrology;

AIMetrology;

Image Processing Aspectof Transverse Shift
Measurement inthe Spin Hall Effect ofLight (SHEL)
Ellipsometry

Ghost Imaging
Microscopy enhanced
by Deep Learning

1

Email: takaya@mech.eng.osaka-u.ac.jp

431

Tool Edge Measurement
by Detecting Laser
Induced Fluorescence ofCutting Fluid

Mitutoyo Association for Science and Technology (MAST); Advisory board
The International Academy for Production Engineering (CIRP); Fellow member, STC-P Vice-chair
International Measurement Confederation (IMEKO); Technical board, TC-14 chair
The Japan Society for Precision Engineering (JSPE) ; Executive bard, Fellow
The Japan Society of Mechanical Engineers (JSME) ; MMT Division Chair, Fellow
The Japan Society for Die and Mould Technology (JSDMT) ; Vice president

2Predictive Production System

Introduction of Research Project
Smart Precision Measurement
Challenge and Innovation

5Challenge and InnovationSmart Precision Measurement

Introduction: Basic scheme & Strategy

NanoSmart Yasuhiro Takaya

Mechanical Engineering, Graduate School of Engineering, Osaka University

123th December 2022

Creation of Future & Innovation: Core Technology from JAPAN

11th Workshop 2022 on Observatory & Measurement Technology from Visible to Infrared

Nano-scale Manufacturing established by Fusion of Optics & Mechanical Engineering!$1.
(Environmental Crisis)[Strategic Metrology in Manufacturing]5234(2"#(Food Crisis)*0-'1(Human originated Disaster)%,)/# 
%,#  Nano diagnostic system%,+&#   Our Laboratory

Biomimetic measurement
and control

Green MEMS: micro
measuring machine system

1.レーザ応用ナノ加工計測技術

Nano measurement in Production,
Design and Integration Course,
Department of Mechanical Engineering,
Graduate School of Engineering,Osaka University

Yasuhiro TAKAYA
NanoSmart@ Kyoto University～ナノスケールものづくり基盤計測技術の確立～

NanoSmart
2.レーザ応用超微細加工技術～ナノ加工技術の確立とナノ計測ツールの開発～

3.ナノ計測システムの製造，組立技術～ナノ生産技術における技術革新～

4.量子光学に基づいた次世代精密計測法～フォトン・メトロロジー基盤技術の確立～

ナノ・マイクロ機械科学 ナノ光科学

非平衡工学

Core-technology of Lab.
NanoSmart Yasuhiro Takaya

Mechanical Engineering, Graduate School of Engineering, Osaka University
23th December 2022

3

New Strategy of Smart Precision Measurement

11th Workshop 2022 on Observatory & Measurement Technology from Visible to Infrared

11th Workshop 2022 on Observatory & Measurement Technology from Visible to Infrared

55

Strategy of Smart Precision Measurement for Predictive production system
2

Robust Measurement

32

41

Cloud Manufacturing1(Metrology IOT/CPS)(Flexible System)

On-machine Measurement ofTool Edge Fine Profile
by Detecting Laser Induced Fluorescence ofCutting Fluid

Environment Adaptive
Metrology

Smart On-machine,In-process Measurement
NetworkPhoton MetrologyAI Metrology

Preciseness ofMeasurement PrincipleIntelligent Measurement(High quality data)(Low Uncertainty)
NanoSmart Yasuhiro Takaya

Mechanical Engineering, Graduate School of Engineering, Osaka University

NanoSmart Yasuhiro Takaya

423th December 2022

Mechanical Engineering, Graduate School of Engineering, Osaka University

11th Workshop 2022 on Observatory & Measurement Technology from Visible to Infrared

PurposeOn-machine measurement method should meet these requirements;

234Tool surface

Exciting-Emitting efficiency [a.u.]

Cutting fluid

2. Reflected light detecting method
Incident lightContamination
by Cutting fluid

ObjectiveTool
reflected
light

Noisy light scattering,Cannot detectfrom the steep angle
surface20.8416nm

1.00.8486nm

Excitation
lightFluorescentemission

0.60.40.6
0.4

Excitation
spectrum

0.2

Fluorescence
spectrum0.0400450500550Oil-based cutting fluid[MP15 (Idemitsu Kosan Co., Ltd.) ]23th December 2022

60.2

Excitation
spectrum0.0350

Wave length [nm]p Using Cutting fluidp Having capability to measure steep angle surface
Mechanical Engineering, Graduate School of Engineering, Osaka University

4

Fluorescence spectrophotometer[F-7000 (Hitachi High-Tech Co., Ltd.)]
398nm 458nm

441nm 457nm

1.0300Development of the on-machine cutting tool profile measurement method
Based on fluorescent confocal microscopy

NanoSmart Yasuhiro Takaya

531

Fluorescence spectrophotometer[UV-3600 (SHIMADZU)]

Conventional on-machine measurement methods and those problems
stylus probe

11th Workshop 2022 on Observatory & Measurement Technology from Visible to Infrared1

ü without cutting fluid disturbanceü steep angle surface
ü in sub- μm precision

523th December 2022Fluorescent emission spectra

5

1. Stylus/probing method

43

NanoSmart Yasuhiro Takaya

600

350

400

Excitation
light
Fluorescentemission
Fluorescence
spectrum
450

500

550600

Wave length [nm]Water-soluble cutting fluid[CFS-100PA (NEOS Co., Ltd.) ]Mechanical Engineering, Graduate School of Engineering, Osaka University

23th December 20227

11th Workshop 2022 on Observatory & Measurement Technology from Visible to Infrared

Proposal: Confocal Fluorescence Microscopy

Confocal fluorescence microscopy

5243Tool surface detection using fluorescence from cutting fluid
ü without cutting fluid disturbanceü steep angle surface
ü in sub- μm precision
441nm

Intensity########a.u.a.u.
Intensity

1.0
1.0#
0.8
0.8#

0.2
0.2#

Excitation
light
Fluorescentemission

0.0
0.0#
300
300#

350
350#

0.4
0.4#

400
400#

Confocal microscope

1486nmLens
Fluorescentemission

y450
500
550
450#
500#
550#
Wavelength nm nm
wavelength#

z600
600#

Roughsurface

Steep angle
surfaceTool

Mechanical Engineering, Graduate School of Engineering, Osaka University

Position coordinate of z-axis μm

10Chamferedface
Crater abrasion-20Confocal fluorescence microscopy
Stylus profilometer
Surface of cutting fluid layer
Confocal laser scanning microscopy-3000100100

1200200300300400400Position coordinate of x-axis

341-40-500500
5000μmImage Processing Aspect ofTransverse Shift Measurement inthe Spin Hall Effect of Light (SHEL) Ellipsometry-10-30-50

52Max. depth: 30.7μm0-20-40

2923th December 2022

11th Workshop 2022 on Observatory & Measurement Technology from Visible to Infrared

4310-10

Mechanical Engineering, Graduate School of Engineering, Osaka University20Max. Depth: 4.7 μm

0Tool surface

Ø Both the lateral andvertical theoreticalresolutions are of submicron order.

5Problem in measurement using the confocal laser scanning microscopy(λ=405 nm) with the cutting fluid layer is suggested.
Shallow abrasion
Deep abrasion
Rake faceyxz
NanoSmart Yasuhiro Takaya

11th Workshop 2022 on Observatory & Measurement Technology from Visible to Infrared20

Ø Only the fluorescenceemitted at the focal
position of the excitation
light can be detected as“a fluorescence signal”.

ObjectivelensTool surface

8

23th December 2022

Comparison with conventional method

1Lens

Cutting fluid layer

Sensing fluorescent emission from cutting fluidwhich is excited by a focused laser light, theposition of a tool surface covered by a cutting
fluid layer can be detected.

The excitation and fluorescent emission spectra ofcutting fluid (MP15) : UV-VIS-NIR Spectrophotometer(UV-3600; Shimadzu Corporation)
NanoSmart Yasuhiro Takaya

Smoothsurface

4

Excitation light Ø Confocal microscope canmeasure the focalFluorescence
position of the excitation
laser light.

ExcitationlightDichroicmirrorx2Scanning

PinholeExcitation
lightCutting
fluid layer

53Detector

Detecting fluorescence
Objective lens

457nm

416nm

0.60.6#11th Workshop 2022 on Observatory & Measurement Technology from Visible to Infrared

Crater abrasion
100
100

200
200

300
300

400
400

Position coordinate of x-axis

500
500μm

The features of the wear with the cutting fluid layer could be evaluatedusing the proposed confocal fluorescence microscopy.
NanoSmart Yasuhiro Takaya

Mechanical Engineering, Graduate School of Engineering, Osaka University

23th December 2022

10NanoSmart Yasuhiro Takaya

Mechanical Engineering, Graduate School of Engineering, Osaka University

23th December 20221111th Workshop 2022 on Observatory & Measurement Technology from Visible to Infrared

Surface Roughness Measurement in Progressive
Semiconductor Industry• Growing interest in device’s scaling down insemiconductor industry
e.g: Rough surface increase scattering andreduce reflectivity of mirror for EUV
lithography• Inspection and measurement demand also
scaled down to sub-nm resolution

Measurement
methodNondestructive tosampleTime

Other Limitation

SEM✕〇

Limited sample typeAFM〇(tapping mode)✕(contact mode)✕Tip variation error,
long measuring time〇〇Mechanicalpolarization
modulator

Spectroscopic
Ellipsometry

Transverse Shift on SHEL Ellipsometry

524311th Workshop 2022 on Observatory & Measurement Technology from Visible to Infrared

Detection Arm
Principle of transverse
shift detection

Incident Arm

Measurement
1Requirement52431

ü Sub-nm resolution
ü Noncontact
ü High-speed

Spin Hall Effect of Light• SHEL shift subject to change depends on

RecordedIntensityz

Recorded SHEL at various (medium properties.
Reflected beams!!
• Spin Hall Effect of Light (SHEL) observation!!
with weak measurement amplifies the shift

SHEL Ellipsometry model:
2& 1 + )() *+,- *+,.* /%""#,%&' =1 + )() /% + 16& % tan% - tan% .*& open the possibility for precision
measurement. e.g surface measurement.

Localization of recorded
beams:• Circle Detection: beam
shape
Surface from Effective Medium Approx.
• Peak Detection: beam5),+ + 5%,+ 6 1 + 5),, 5%,, 6
profile4=→ 8+ = & ) Detection of the transverse shift5),, + 5%,, 6 1 + 5),+ 5%,+ 6
in order to solve the formula to!:exp −4'()$ cos-# ./0
obtain surface as a function ofd: thickness of roughness layer (( = 1.5 '/2CD + 0.4 nm )ellipsometry parameter
I$ , I% :amplitude reflection coefficient of air-film and film-substrate" , 2$% ∶ '()*+,+-% .ℎ+,0 ;2: -**+).4(-056 )'5'(-0-5;!!
,: *-$. 2 ,478.; 9# : +$7+%-$0 '$:*-; ;: <-'( =+%0ℎ• Transversal shift ("! )measurement in SHEL
has been rarely discussed""

Incident
BeamyNanoSmart Yasuhiro TakayaxOptions on transversal shift measurements
are explored as the preliminary studies toevaluate the accuracy

Mechanical Engineering, Graduate School of Engineering, Osaka University

1223th December 2022

NanoSmart Yasuhiro Takaya

Mechanical Engineering, Graduate School of Engineering, Osaka University

11th Workshop 2022 on Observatory & Measurement Technology from Visible to Infrared

Transverse Shift Measurement
Circle Detection

Peak Detection

Thresholding/
Binerization

Max. intensity
localization,
row data
extraxtion

CircleEstimation

Gaussianfitting

CenterDetection

NanoSmart Yasuhiro Takaya

34Circle Detection

152

Peak Detection"" = 232.5 <=341"" = 242.7 <=

221.2

229.9

215.0

228.7

243.8

280.8

• Extreme points happening in the same spot at both approaches potentiallycan be considered as object’s feature• White outliers, fail detection by each methods not due to object’s feature

PeakDetection

Mechanical Engineering, Graduate School of Engineering, Osaka University

11th Workshop 2022 on Observatory & Measurement Technology from Visible to Infrared

Mapping of detected amplified transversal shift

5213

23th December 2022

23th December 2022

14NanoSmart Yasuhiro TakayaMechanical Engineering, Graduate School of Engineering, Osaka University

23th December 202215

11th Workshop 2022 on Observatory & Measurement Technology from Visible to Infrared

Plot Comparison

11th Workshop 2022 on Observatory & Measurement Technology from Visible to Infrared

5254

32

1341Ghost Imaging Microscopy enhanced byDeep Learning"" = 242.7 ± 6.7 <="" = 232.5 ± 4.3 <=•Considering that this result acts as feed to obtain surface properties, it shows how
important the selection of shift detection method to the accuracy of measurement

NanoSmart Yasuhiro TakayaMechanical Engineering, Graduate School of Engineering, Osaka University

16

23th December 2022

NanoSmart Yasuhiro Takaya

Mechanical Engineering, Graduate School of Engineering, Osaka University

11th Workshop 2022 on Observatory & Measurement Technology from Visible to Infrared

What is Ghost Imaging ?

3

11th Workshop 2022 on Observatory & Measurement Technology from Visible to Infrared

Principle of Ghost Imaging

52

4

17

23th December 2022

52

1

3

4

1Imaging by second-order correlation of fluctuation

imageintensity fluctuation
oftime dependent signal

intensity fluctuation
ofilluminationAPPLICATION fields; Quantum optics, Medical imaging,
Optical astronomy, Industrial inspection, etc.
NanoSmart Yasuhiro Takaya

Mechanical Engineering, Graduate School of Engineering, Osaka University

23th December 2022

18

NanoSmart Yasuhiro Takaya

Mechanical Engineering, Graduate School of Engineering, Osaka University

23th December 2022

19

11th Workshop 2022 on Observatory & Measurement Technology from Visible to Infrared

Basic concept of GI with Deep Learning

5

Basic configuration of Ghost Imaging

32

11th Workshop 2022 on Observatory & Measurement Technology from Visible to Infrared

4illumination noise(fluctuation of atmosphere,
light and sample)

1

52

3

4

1
detector noise(thermal and shot noise)

Ghost imaging

scattered light, fluorescence, reflectance, transmittancemulti-wavelength…

timeFewer measurements= Low quality

NanoSmart Yasuhiro Takaya

Mechanical Engineering, Graduate School of Engineering, Osaka University

20

23th December 2022

NanoSmart Yasuhiro Takaya

High quality
Mechanical Engineering, Graduate School of Engineering, Osaka University

11th Workshop 2022 on Observatory & Measurement Technology from Visible to Infrared

Experimental results for GI with Deep Learning
ROI [μm2]

250 × 250Image size[pixel]

28×28

Number of GI meas.

5DMD Frequency [Hz]60

Epoch N200

4Glass micro beards(f=10um)

Sample image
1(detected by microscope)

Low image quality(almost invisible)
100 μmDetected GI images

5 measurements, 200 epoch

3908 measurements
1.0

Number of GI meas.

5DMD Frequency [Hz]60

Epoch N200

Mechanical Engineering, Graduate School of Engineering, Osaka University

23th December 2022

1Ox

Sample was moved by using manually operated
stage with micrometer.

With deep learning
5 measurements, 200 epoch1.0

1.00022NanoSmart Yasuhiro Takaya

4% ", $ [a.u.]

CNR=16.21

3y! ", $ [a.u.]

028×28% ", $ [a.u.]

CNR=16.21

250 × 250

1.0! ", $ [a.u.]! ", $ [a.u.]

0ROI [μm2]
Image size[pixel]

Detected GI images
5 measurements

With deep learning

1.0

NanoSmart Yasuhiro Takaya

52Illumination area

experimental results

More than 800 times as fast as GI imaging
5 measurements

11th Workshop 2022 on Observatory & Measurement Technology from Visible to Infrared

experimental conditions

experimental conditions

2123th December 2022

Imaging results of moving sample

532Deep learning

Convolutional Neural Network: CNN

0100 μm

Mechanical Engineering, Graduate School of Engineering, Osaka University

23th December 202223

11th Workshop 2022 on Observatory & Measurement Technology from Visible to Infrared

525

Smart On-machine measurement Strategy using
Multi-sensor system

43

11th Workshop 2022 on Observatory & Measurement Technology from Visible to Infrared14321

マルチセンサ融合戦略：異なる物理量の特徴を活用し検知能力を向上

Predictive Production System

電流計

マルチセンサ協調戦略：同じ幾何学量の測定を目的とする構成要素を最適化

位置検出プローブ

工具
光検出器
光センサ

モータ電流

レーザ変位センサ

熱変位工具
音波AEセンサ

振動

加工状態の監視［加工異常，工具摩耗・折損，動的特性など］

発光

幾何学量の高精度計測

温度

加速度

［寸法，形状，位置，姿勢，表面性状，変位，運動精度など］

レーザトラッカ

応力工具ਃ঱એ႗২ੑ೾‫ق‬ছইॿ५।‫ش‬४‫ك‬ਃ঱दએ႗২ଐ౯ਖ਼৒ऋ૭ચ
ਃ༊೿ૃਂਏ‫ؚ‬ড‫ش‬ॡ॑਄ਗऔङप೾৒૭ચ

Ω
ひずみゲージレーザ干渉測長機ઃ੪ড‫ش‬ॡੑ೾ලಔએ႗২ੑ

ছইॿ५।‫ش‬४

̭P

̭P

ලಔએ႗২ੑ

ල౐प‫گ‬ઃ੪ঽ૓ශએ॑ੑ೾

工具Ω

加速度計

5*એ႗২ੑ

校正用ゲージ

熱電対

5D̭P

表面粗さスタイラス
5D̭P

3738)RUP&RQWUROभ્শMulti-sensor Fusionय़কজঈঞ‫ش‬३ঙথ

Multi-sensor Cooperationල౐ऩੑ೾ਡभ੐ં ড‫ش‬ॡभ'ঔॹঝ॑નੳखऩऋै೾৒ᆽਚ॑ભਔप੐৒ ਃ঱द೾৒भ୔खःঽ૓ශએभ೾৒ु૭ચ੗ਡय़কজঈঞ‫ش‬३ঙথपेॊৈಖ২ऩએઉੑ೾ॱॵॳউট‫ش‬ঈभ஘‫ر‬ऩ্਱षभ౵ो୤॑ଓਫ
೷ନखಖ২

‫̭ڱ‬Pਰৣ

೾৒৽ଡ଼भঽ৿ৠ৒

NanoSmart Yasuhiro Takaya

Mechanical Engineering, Graduate School of Engineering, Osaka University24

23th December 2022

NanoSmart Yasuhiro Takaya

Mechanical Engineering, Graduate School of Engineering, Osaka University ੑ೾উটॢছ঒म‫ؚ‬३५ॸ঒ऋঽ৿ੑ઴ ๝௱ॳख़ॵॡु3&঱द૭ચ

23th December 2022

25

39

11th Workshop 2022 on Observatory & Measurement Technology from Visible to Infrared

Smart measurement system & machine tool 2for Predictive Production System
Machine monitoring by smart precision measurement
in cloud manufacturing

IoT

マシン・モニタリング

IoT

サイバー・
フィジカル・システム
インダストリアル・
ビッグデータPredictive Production System
生産システムE

クラウド

予測モデル

IoT

生産システムB

5

Thank you for your attention !

マシン・コンディション・モニタリング

IoT
IoT

クラウド

データ駆動解析

物理モデル解析

CMIoT＊CM : クラウド・マニュファクチャリング
生産品質・ループ

IoT

ホリスティック計測

グローバル・ループ

・・・
スマート計測機

・ 機内・機上計測・ インプロセス計測

生産・ループ

23

4

Environment
Adaptive
Metrology;

PhotonMetrology;

AI
Metrology;

Image Processing Aspectof Transverse Shift
Measurement inthe Spin Hall Effect ofLight (SHEL)
Ellipsometry

Ghost Imaging
Microscopy enhanced
by Deep Learning

Tool Edge Measurement
by Detecting Laser
Induced Fluorescence ofCutting Fluid（ボリュームデータ融合）

計測・ループローカル・ループ・ メトロロジーX線CT・ マルチセンサCMM

加工・ループ・・・
スマート工作機械

NanoSmart Yasuhiro Takaya

高谷 裕浩

1

生産システムD
生産システムC

大阪大学 大学院工学研究科
機械工学専攻統合設計学系
ナノ加工計測学領域

Q&A

4

プロセス・モニタリング

クラウド

11th Workshop 2022 on Observatory & Measurement Technology from Visible to Infrared

5340

生産システムA

1CAD/CAM
CAE/CAT

スマート工作機械
Mechanical Engineering, Graduate School of Engineering, Osaka University

スマート計測機

23th December 2022

26

Introduction: Basic scheme & Strategy

NanoSmart Yasuhiro Takaya

Mechanical Engineering, Graduate School of Engineering, Osaka University

23th December 2022

27