Charged Particle Beam Apparatus and Report  Creation Method

An input and output device includes: an instruction analysis unit configured to generate a conversation document in which a conversation uttered by a user is converted into character string data and recognize, based on the conversation document, a conversation intention of the user including an instruction to an image acquisition device; a history retention unit configured to record, as history information, the conversation document, the conversation intention, and a response of the image acquisition device to the instruction to the image acquisition device; a difference analysis unit configured to divide a report creation period using a timing when the user issues, to the image acquisition device, an instruction including an intention to save a captured image as a boundary and output report creation information in which history information divided for each of the report creation periods, and a captured image and a differential condition corresponding to the history information are associated with each other; and a report retention unit configured to create a report for each of the report creation periods based on the report creation information and record the created report.

TECHNICAL FIELD

The present invention relates to a charged particle beam apparatus and a report creation method.

BACKGROUND ART

As typified by a translation machine and the like, techniques such as voice recognition and natural language processing are commercialized and expected to be used in various fields. As for a charged particle beam apparatus, experiments that require a beam to be quickly operated, such as processing on a sample that is likely to be damaged by an electron beam or real-time observation, become a trend.

To ensure validity of an observation image, it is necessary to accurately correspond with traceability between a process leading up to imaging and the observation image. Further, after an experiment using a charged particle microscope is conducted, it is generally necessary to submit a report including an observation image. However, a task of organizing a huge amount of information associated with the experiment is a burden on a user.

For example, PTL 1 discloses a technique in which an icon is displayed on an observation image and the icon provides a link to a data object including text and voice recording. Specifically, it is disclosed that more efficient acquisition and analysis of object image data are enabled by embedding and layering the voice recording in the observation image.

PTL 2 discloses a technique in which image data is acquired by extracting a still image from a moving image at a predetermined timing, voice data is acquired by extracting voice for each of speech sections separated by silence sections from voice recorded in synchronization with the moving image, and the image data and the voice data are recorded in association with each other.

CITATION LIST

Patent Literature

SUMMARY OF INVENTION

Technical Problem

In general, an electron microscope device records a captured image and an observation condition in association with each other, whereas it is necessary to manually record, based on an experiment note or the like, a process leading up to imaging of an observation image. However, when the process leading up to imaging is not accurately recorded due to omission or the like, the process becomes ambiguous and data reliability is impaired. Therefore, it may be difficult to reproduce an experiment. Therefore, it is fairly important to accurately record traceability between the process leading up to imaging of the observation image and the observation image in order to ensure validity of the observation image.

In addition, in order to record every step of the process leading up to imaging, for example, there is a method of recording a state of an experiment with video cameras fixed at plural locations. However, when the process is not clear from the state of the experiment alone, an operator needs to explain the state of the experiment. However, according to such a method, a burden on the operator may increase according to recording by the video cameras, management of a large amount of video data, extraction of necessary information from the video data, and the like.

An object of the invention is to provide a charged particle beam apparatus capable of easily ensuring validity of an observation image.

Solution to Problem

A charged particle beam apparatus according to the invention includes an image acquisition device configured to acquire a captured image of a sample and an input and output device configured to operate the image acquisition device. The input and output device includes: an instruction analysis unit configured to generate a conversation document in which a conversation uttered by a user is converted into character string data and recognize, based on the conversation document, a conversation intention of the user including an instruction to the image acquisition device; a history retention unit configured to record, as history information, the conversation document, the conversation intention, and a response of the image acquisition device to the instruction to the image acquisition device; a difference analysis unit configured to divide a report creation period using a timing when the user issues, to the image acquisition device, an instruction including an intention to save the captured image as a boundary, compare an imaging condition at a timing when an instruction including an intention to save a current captured image is issued with an imaging condition at a timing when an instruction including an intention to save a previous captured image is issued, extract a common condition and a differential condition of imaging conditions between a previous report creation period and a current report creation period, and output report creation information in which the history information divided for each of the report creation periods, and the captured image and the differential condition corresponding to the history information are associated with each other; and a report retention unit configured to create a report for each of the report creation periods based on the report creation information, and record the created report.

Advantageous Effects of Invention

According to the invention, it is possible to easily ensure validity of an observation image.

DESCRIPTION OF EMBODIMENTS

Hereinafter, embodiments of the invention will be described in detail with reference to the drawings. In all the drawings for showing the embodiments, the same members are denoted by the same reference numerals in principle, and repeated description thereof is omitted.

First Embodiment

<Configuration of Charged Particle Beam Apparatus>

FIG.1shows a configuration example of a charged particle beam apparatus according to a first embodiment of the invention. A charged particle beam apparatus100according to the embodiment is a device that emits a charged particle beam to a sample. Hereinafter, a device that acquires an observation image of a sample by emitting a charged particle beam to the sample, such as a scanning electron microscope (SEM), will be described as an example. In addition, the embodiment is assumed to be used by a plurality of persons. Here, a case where a first user101and a second user102use the device will be described as an example, and the device is usable by three or more users.

The charged particle beam apparatus100includes an image acquisition device103, a first input and output device110, a second input and output device130, and the like. The first input and output device110is a device used by the first user101, and the second input and output device130is a device used by the second user102. The image acquisition device103, the first input and output device110, and the second input and output device130are connected by wire, wirelessly, or via a network (for example, the Internet or an internal network), and can transmit and receive various types of information therebetween, such as a conversation document, a conversation intention, a control signal, a captured image, and an imaging condition to be described later. Hereinafter, the first input and output device110and the second input and output device130may be referred to as input and output devices.

The image acquisition device103includes a charged particle gun (for example, an electron gun) that emits charged particles (for example, electrons) to a sample, a lens that controls a trajectory of the charged particles, a stage on which the sample is placed, a detector that detects secondary particles and the like emitted from the sample after the charged particles are emitted thereto, a communication interface that performs communication with the input and output device, and the like (all not shown). The image acquisition device103emits the charged particles from the charged particle gun based on a control signal received from the input and output device and detects, by the detector, the secondary particles emitted from the sample. The image acquisition device103generates an observation image of the sample based on the secondary electrons detected by the detector. The image acquisition device103transmits the generated observation image to the input and output device. The image acquisition device103may store the generated observation image in a storage device (not shown) in the image acquisition device103.

The first input and output device110and the second input and output device130have the same configuration. Therefore, the first input and output device110will be described as an example to explain a configuration of the input and output device.

The first input and output device110receives an input operation performed by the first user101and controls the image acquisition device103by transmitting a control signal based on the input operation. The first input and output device110receives, from the image acquisition device103, various types of information such as the observation image generated by the image acquisition device103and state data indicating a state of the image acquisition device103. The first input and output device110shares the observation image and a process leading up to imaging of the observation image with the second input and output device130.

As shown inFIG.1, the first input and output device110includes a speaker111, a microphone112, an instruction analysis unit113, a control unit114, a history retention unit115, a difference analysis unit116, a report retention unit119, and a monitor120.

The first input and output device110includes the microphone112, the speaker111, and the monitor120which are man-machine interfaces. The first input and output device110requires a computer such as a CPU that implements calculation of the instruction analysis unit113and the difference analysis unit116, a storage device that implements functions of the history retention unit115and the report retention unit119, and a communication interface that transmits and receives information to and from the image acquisition device103and an input and output device of another user participating in an experiment. Therefore, as the input and output device, for example, an information processing terminal such as a smartphone, a tablet terminal, or a personal computer (PC) can be used. When a software program is executed by a computer of the information processing terminal, functional blocks such as the instruction analysis unit113and the difference analysis unit116are implemented.

In addition, the input and output device110used by the first user101and the input and output device130used by the second user102have the same function. However, it is necessary to perform time synchronization between these devices.

The microphone112acquires voice instruction issued to the charged particle beam apparatus100by the first user101and voice of discussion between the first user101and other users including the second user102. The microphone112converts the acquired voice into voice data and outputs the voice data to the instruction analysis unit115. In this way, information including thinking of the user is input to the charged particle beam apparatus100as the voice data.

An input device121is a device such as a keyboard or a mouse through which a document is inputted by an operation of the user. The input device121may be, for example, a software keyboard displayed on the monitor120. The software keyboard may be operated by a mouse or may be operated by the monitor120having a function as a touch panel. For example, the first user101can perform an input operation by inputting a document to an instruction input area215by using a keyboard and clicking a transmission button216by using a mouse. In this way, the user can perform an input operation to the input and output device100by both inputting voice via the microphone112and inputting text data or the like by the input device121.

The instruction analysis unit113analyzes a conversation uttered by the user received from the microphone112, and extracts, from the conversation, an instruction content for the image acquisition device103. The instruction analysis unit113performs language understanding on the voice data received from the microphone112. Specifically, by using a deep learning model (first deep learning model) trained by a pair of a learning conversation (voice data) and a learning document (character string data) that is a document of the conversation, the instruction analysis unit113generates a conversation document as character string data based on a conversation (voice data) of the first user101. The deep learning model used here is, for example, a long-short term memory (LSTM), and is not limited thereto.

Next, by using a deep learning model (second deep learning model), for example, a recurrent neural network (RNN) trained by a pair of the learning document (character string data) and a learning intention that is a conversation intention corresponding to the learning document, the instruction analysis unit113recognizes and generates a conversation intention of the first user101based on the conversation document generated by the deep learning model. In this way, the instruction analysis unit113performs language understanding on the voice data and outputs the conversation document and the conversation intention to the control unit114and the history retention unit115.

The control unit114controls the image acquisition device103based on a processing result of the language understanding of the voice data in the instruction analysis unit113. The control unit114generates a control signal of the image acquisition device103according to the conversation intention of the first user101received from the instruction analysis unit113and transmits the generated control signal to the image acquisition device103.

The control unit114receives a response signal from the image acquisition device103as a control result based on the control signal. The control unit114outputs the received response signal to the speaker111, the history retention unit115, the difference analysis unit116, and the report retention unit119. The control unit114may generate a signal based on the response signal and output the generated signal.

A content of the response signal is conveyed from the speaker111to the first user101by voice. Accordingly, for example, the first user101can check whether the content instructed by the conversation is accurately conveyed to the image acquisition device103. In addition, even when an instruction is repeatedly issued to the image acquisition device103, the first user101can check whether a content of the instruction is conveyed each time the instruction is issued, and can perform a response operation. In addition, as will be described later, the control unit114may display, in terms of text on the monitor, a conversation of each user and a response of the image acquisition device103to an instruction of the user.

In addition, the control unit114transmits the conversation document and the conversation intention of the conversation uttered by the first user101to an input and output device of another user participating in the experiment. Similarly, the control unit114receives a conversation document and a conversation intention of a conversation uttered by another user from an input and output device of the user who utters the conversation. The control unit114outputs the conversation document and the conversation intention of the other user to the history retention unit115.

The control unit114also has a function as a communication interface that transmits and receives information between the first input and output device110and the image acquisition device103. In addition, the control unit114also has a function as a communication interface that transmits and receives information between the first input and output device110and the second input and output device130. The function of the communication interface may also be provided separately from the control unit114. The image acquisition device103and the second input and output device130may use different communication interfaces.

The control unit114implements each functional block of the first input and output device110by executing a software program on a computer such as a CPU. Alternatively, a part of functional blocks of the input and output device110may be implemented by hardware.

The history retention unit115records, as history information, the conversation document and the conversation intention of the first user101received from the instruction analysis unit113, a response (for example, a response signal in a command format) received from the control unit114, a conversation document and a conversation intention of another user, and the like. At this time, the history retention unit115records the conversation document, the conversation intention, and the response signal in association with a user ID of the user who utters the conversation and time when the conversation is uttered. That is, the history information includes the user ID of the user who utters the conversation and the time when the conversation is uttered.

The history retention unit115and the report retention unit119include a storage device. The storage device may be a non-volatile memory such as a flash memory or a volatile memory such as an SRAM or a DRAM.

The difference analysis unit116divides a report creation period using a timing when the user issues an instruction including an intention to save a captured image as a boundary. The history information recorded in the history retention unit115, a captured image117and an imaging condition118output from the control unit114are input to the difference analysis unit116. The difference analysis unit116compares an imaging condition in a current report creation period with an imaging condition in a previous report creation period and extracts a common condition and a differential condition of the imaging conditions between the previous report creation period and the current report creation period. The difference analysis unit116outputs, as report creation information to the report retention unit119, information in which history information divided for each of the report creation periods, the captured image117and the differential condition corresponding to the history information, and the like are associated with each other. The common conditions of the imaging conditions are separately output to the report retention unit119.

The captured image117is a plurality of images saved by the image acquisition device103. The imaging condition118is a condition such as a control value set in the image acquisition device103when the image acquisition device103saves the captured image117or a measurement value obtained from a sensor.

The report retention unit119creates and records a report using the report creation information and the common condition of the imaging conditions received from the difference analysis unit116. At this time, it is desirable that the report retention unit119records the report after converting the report into a format that can be viewed on the monitor120or the like such that the report is smoothly displayed when there is a report writing instruction to be described later. In addition, the report retention unit119also records back data in association with the report creation information such that displaying and hiding of each item in the report can be dynamically switched.

The first user101can view the report that is output to and displayed on the monitor120of the first input and output device110. In addition, the report may also be output to and printed by a printer that can print on a paper medium.

<Operation Screen of Input and Output Device>

Next, an operation screen displayed on the input and output device will be described. In addition, an operation example of the input and output device and the image acquisition device will also be described.

FIG.2shows an example of the operation screen of the input and output device inFIG.1. Here, an operation screen displayed on the first input and output device110will be mainly described, and a case where the operation screen is applied to the second input and output device130may also be described as necessary.

As shown inFIG.2, an operation screen200is displayed on the monitor120, and includes a conversation display area201, the instruction input area215, the transmission button216, an observation image display area220, a condition display area221, a log-in and log-out button230, a mute button231, a report writing button232, and a report reading button234.

In the conversation display area201, text corresponding to a conversation (conversation document) of the user of the charged particle beam apparatus100and a response (response signal) from the image acquisition device103in response to an instruction of the user are displayed.

When there is voice received from the first user101, an account name202of the first user101and a call-out203of an input conversation of the first user101are associated with each other and displayed as one set in the conversation display area201. A content of the call-out203is text display of the conversation document of the first user101output from the instruction analysis unit113.

The user participating in the experiment issues an instruction to the image acquisition device103by a conversation while referring to an observation image displayed in the observation image display area220and an imaging condition of a captured image displayed in the condition display area221, which will be described later.

In the example inFIG.2, the conversation of the call-out203is “align the optical axis”. The instruction analysis unit113generates the conversation document of the first user101based on the voice received from the microphone112, and outputs the generated conversation document to the control unit114. The control unit114outputs the input conversation document to the monitor120, and the monitor displays the input conversation document on a left side in the conversation display area201of the operation screen200.

In addition, the control unit114transmits the input conversation document to the second input and output device130. A control unit of the second input and output device130outputs the received conversation document of the first user101to a monitor, and the monitor displays the input conversation document in a conversation display area of an operation screen. In this way, conversations of all the users are displayed in real-time on input and output devices of participants participating in the experiment, and thus the instruction content issued to the image acquisition device130can be shared.

In addition, the image acquisition device103transmits the response signal not only to the first input and output device110but also to the input and output devices of all the users participating in the experiment. Accordingly, the response content of the image acquisition device103in response to the instruction of the user is displayed on the input and output devices of all the users.

The account name202displays an ID for specifying the first user101in a character string, and for example, “participant A” is displayed inFIG.2. In addition to the account name202and the call-out203, time when the voice is input may be displayed together.

In addition, based on the conversation “align the optical axis” in the call-out203, the instruction analysis unit113recognizes that the conversation is an instruction issued to the image acquisition device103, extracts a conversation intention of the conversation, and outputs the extracted conversation intention to the control unit114. The control unit114transmits, to the image acquisition device103, a control signal generated based on the conversation intention received from the instruction analysis unit113.

A call-out204inFIG.2is text of a response from the image acquisition device103in response to the instruction “align the optical axis” from the first user101. The image acquisition device103transmits a response signal in response to the control signal to the control unit114of the first input and output device110. The control unit114outputs the received response signal to the monitor120, and the monitor displays text corresponding to the input response signal as the response to the instruction of the user on a right side in the conversation display area201.

In addition, the image acquisition device103transmits the response signal not only to the first input and output device110but also to the input and output devices of all the users participating in the experiment. Accordingly, the response content of the image acquisition device103in response to the instruction of each user is displayed on the input and output devices of all the users.

InFIG.2, “voice operation system” is displayed as an account name corresponding to the call-out204, and any other expression may be used as long as the user can recognize the response of the image acquisition device103.

The response in the call-out204is “perform alignment”. In this way, the image acquisition device103responds to the instruction “align the optical axis” of the first user101by performing “alignment” of the optical axis. Thereafter, although not shown inFIG.2, the image acquisition device103performs alignment of the optical axis and transmits a captured image after the alignment to the input and output devices of all the users.

Thereafter, the first user101instructs the image acquisition device103to capture an image at an increased magnification by a conversation in a call-out205. The image acquisition device103responds by capturing an image at an increased magnification, and this response is displayed as a call-out206. The image acquisition device103transmits the captured image captured at the changed magnification and a changed imaging condition to each input and output device. Then, a report is created at this timing. The report includes the conversations in the call-outs203to206, conversation intentions corresponding to the instructions issued by the conversations in the call-outs203and205, the captured image acquired based on the instruction of the conversation in the call-out205, the imaging condition, a differential condition and a common condition at that time, and the like.

While the first user101and the second user102view the captured image displayed on the respective input and output devices, the first user101inquires of the second user102, through a conversation in a call-out207, about the newly captured image. Since the conversation “how is it?” in the call-out207is not an instruction issued to the image acquisition device103, the instruction analysis unit113only creates a conversation document and does not create any conversation intention.

In addition, in response to the conversation in the call-out207, the second user102expresses a view that “it is desirable to make the internal structure slightly clearer” in a conversation in a call-out208. The conversation in the call-out208is “it is desirable to make the internal structure slightly clearer”, which does not issue any instruction to the image acquisition device103, so that an instruction analysis unit of the second input and output device130only creates a conversation document and does not create any conversation intention.

Then, the second user102issues an instruction “adjust brightness and save again” in a conversation in a call-out209. The instruction analysis unit of the second input and output device130creates and outputs a conversation document and a conversation intention corresponding to the conversation in the call-out209.

In response to the instruction issued by the conversation in the call-out209, the image acquisition device responds in a response in a call-out210that brightness is adjusted to capture an image. Then, the image acquisition device103acquires a captured image whose brightness is adjusted and transmits the captured image and an imaging condition thereof to each input and output device.

Then, a report is created again at this timing. The report includes the conversations in the call-outs207to210, a conversation intention corresponding to the instruction issued by the conversation in the call-out209, the captured image acquired based on the instruction of the conversation in the call-out209, the imaging condition, a differential condition and a common condition at that time, and the like.

Although a case where the conversation document and the conversation intention are created based on the conversation uttered by the user is described so far, a document may also be input into the instruction input area215by using the input device121(may be implemented by software) such as a keyboard, and the input may be made by voice from the user by an instruction using the transmission button216. In this case, the input document input by the user is the conversation document, and the instruction analysis unit113creates a conversation intention based on the input document. That is, in this case, the instruction analysis unit113may not create any conversation document.

In the observation image display area220, the captured image acquired by the image acquisition device103is displayed as an observation image in real-time. The first user101and the second user102make a conversation between users and instruct the image acquisition device103while referring to the observation image.

In the condition display area221, an imaging condition of a captured image set in the image acquisition device103, that is, an observation condition of the sample is displayed. In the condition display area221, conditions such as a control value or a measurement value obtained from a sensor are displayed as shown inFIG.2as the imaging condition of the captured image. The first user101or the second user102determines a condition to be set next based on these conditions and issues an instruction to the image acquisition device by voice or by document received from the input device.

As shown inFIG.2, the imaging condition includes, for example, an imaging magnification, an acceleration voltage, a working distance (WD), a detection signal, and a stage position (X, Y). Here, examples of the detection signal include secondary electrons (SE), reflected electrons, X-ray, Auger electrons, analysis contrast, Z contrast, and a focused ion beam.

The log-in and log-out button230is a button for opening a window so as to input the user ID. When the user ID is input, input of a password may be requested as necessary. In addition, when a plurality of users participate in the experiment, linkage information for linking the input and output devices of the users may be input into the window. Examples of the linkage information include a unique conference name and a list of connection destination IP addresses managed by a certain server.

The mute button231is a button for turning off acquisition of voice signals. By providing the mute button231, it is possible to prevent a voice signal irrelevant to the experiment from being processed. In this case, the button is turned on only when no conversation is to be input. In addition, the mute button231may be changed to a button for starting input of a conversation. In this case, this button is turned on only when a conversation is to be input.

In addition, like a speaker (AI speaker) using artificial intelligence, on and off of conversation input may be switched by a key word. In this case, the mute button231or the button for starting conversation input may still be provided.

The report writing button232is a button for outputting a report recorded in the report retention unit119. When the report writing button232is pressed, the report is displayed on the monitor120in a viewable format. In addition, the report may also be printed from a printer. Since reports are recorded in the report retention unit119in a viewable format, a burden of creating reports and collecting information from a large number of reports routinely performed by the user is reduced.

The report reading button234is a button used for conducting an experiment using an already generated report. When the report reading button234is pressed, the report recorded in the report retention unit119is read. In addition, when the report is to be read, a list of reports may be displayed on the monitor120first, and only a report selected by the user from the list may be read.

For example, the control unit114extracts latest imaging conditions among common conditions and differential conditions in the report selected by the user, and collectively sets the extracted imaging conditions in the image acquisition device103. In addition, a differential condition may be appropriately selected from a plurality of conditions in the report. Accordingly, even when the experiment is temporarily interrupted, the experiment can be resumed under the same conditions as long as a report at that time is retained.

In addition, when validity of data described in the report is questioned, the report is read by pressing the report reading button234, and it is possible to perform re-observation under the same imaging condition by referring to the read report. Accordingly, if the same observation image is obtained, the validity of the data can be verified. Alternatively, when the validity of the data cannot be determined due to data insufficiency, it is possible to acquire the data later.

<Example of Created Report>

Next, an example of a report created in the embodiment will be described.FIG.3shows an example of the report corresponding toFIG.2. A report REP inFIG.3includes a title description area301, a comment insertion button302, a save button303, a common condition description area310, a first data set description area320, a second data set description area330, and the like.

In the title description area301, an experiment title input by the user is displayed. In addition, the user can add information for specifying the experiment as necessary, such as experiment date and time. Accordingly, readability and searchability of an experiment report can be improved.

The comment insertion button302is a button used to insert text or voice data into the report REP for the purpose of adding an item insufficient in the experiment, correcting an error, or the like.

The save button303is a button used, when an output report is edited, to save the edited report in the report retention unit119. Accordingly, even when data is organized after the experiment, the report can be centrally managed in the report retention unit119.

The common condition description area310is an area in which a common condition among imaging conditions is displayed. In the common condition description area310, items in the common condition of the report are displayed. However, an item to be displayed as the common condition may be selected for each experiment by pressing a common condition edition button311. For example, as for a sample on which EDS analysis is performed for elemental analysis, unlike a sample on which no EDS analysis is performed, an emission current may be displayed as the common condition.

In the first data set description area320, a first captured image321is displayed. In addition, a first differential condition322extracted by the difference analysis unit116is displayed in the vicinity of the first captured image321(immediately below the captured image inFIG.3) as information describing a difference between the first captured image321and other captured images. A first delete button323is displayed on the first captured image321. When the first delete button323is pressed, the first data set description area320is deleted from the report. At this time, a part of display contents may be integrated with the second data set description area330.

Further, in the first data set description area320, a first imaging process324is displayed, which shows a list of intentions, details, and procedures through which the first captured image321is acquired. The first imaging process324shows a history divided by the difference analysis unit116using a timing when the captured image321is saved as a boundary. For example, as shown inFIG.3, the first imaging process324includes time when a conversation is uttered (recorded), a user ID of a user who utters the conversation, a conversation document, and an operation performed on the image acquisition device103based on the conversation.

In addition, as the first imaging process324, items such as a response of the image acquisition device103, an intermediate output of the instruction analysis unit or the difference analysis unit in each input and output device, and a desired measurement value may be displayed in addition to those described above.

Similarly to the first data set description area320, in the second data set description area330, a second captured image331, a second differential condition332, and a second imaging process334are displayed. A second delete button333is displayed on the second captured image331.

According to the report REP inFIG.3, in an experiment conducted on month XX day YY, in year ZZZZ, two images with different contrasts in the same field of view are captured at a magnification of 30,000 times. According to the report inFIG.3, a participant A and a participant B participate in the experiment, and it is also possible to read an intention of brightness adjustment for improving a contrast of an internal structure and a detail that the participant B is not satisfied with quality of the first captured image321. When the report does not satisfy a predetermined requirement and imaging is performed again, contents in the first data set description area320may not be described in the report.

When the first delete button323is pressed, the first captured image321and the first differential condition322are deleted from the report REP. The first imaging process324may be integrated into the second imaging process334since a process leading up to obtainment of the second captured image331is described therein.

In addition, “contrast: +1” displayed in the second differential condition332is obtained by extracting an imaging condition changed between the first captured image321and the second captured image331. Therefore, when the first data set description area320is deleted, the changed imaging condition is not extracted, and thus the second differential condition332may be blank.

<Search for Process Leading Up to Imaging>

FIG.4shows a method of searching for a process leading up to imaging. A search screen400shown inFIG.4includes a search content input section401, a search button402, an imaging process403, a highlight404, a captured image405, a selection button406, an OK button407, and a cancel button408.

Here, the participant A (the first user101) participates in an experiment of observing a sample A, a sample B, and a sample C and has no knowledge about how to set an imaging condition for a sample D to be observed for the first time, and thus it is assumed that the imaging condition is set by using a function of searching for a process leading up to imaging.

First, the participant inputs, into the user search content input section401, “observe an internal structure of a plating process in a milled cross section of sample D”, which is an observation purpose, and presses the search button402.

Since a conversation document recorded in the report retention unit119includes thinking leading up to obtainment of a target observation image and contents of discussion, for example, the control unit114searches reports recorded in the report retention unit119by using the observation purpose input here as a search key word, and extracts a sentence similar to the search key word.

On the search screen400, for example, the imaging process403including a conversation document matching with or similar to the search key word is displayed as a search result. In addition, in the imaging process403, a portion that matches with or is similar to the search key word is displayed by the highlight404. InFIG.4, the highlight404is indicated by hatching, and the highlight404can also be implemented by using a conspicuous color instead of hatching. On the search screen400, the captured image405corresponding to the imaging process403extracted in the search is displayed side by side with the imaging process403.

InFIG.4, data sets of three reports extracted by the search are listed. Among these data sets, the participant A selects, by the selection button406, a data set determined to be appropriate for the sample D to be observed for the first time, and presses the OK button407. Then, the control unit114acquires, from the report retention unit119, a common condition and a differential condition in the selected data set, and sets the acquired conditions in the image acquisition device103.

When search results do not include any data set that can be determined to be appropriate, by pressing the cancel button408, it is possible to start the observation of the sample D without changing conditions in the image acquisition device103.

The search results inFIG.4all show examples in which an imaging condition is satisfactorily set to capture an image of an internal structure. Among these search results, the participant A determines that a second condition from the top for capturing an image of an inside structure of a hole of the same metal as the plating process is appropriate, selects a second data set from the top, and starts observation under conditions under which a captured image corresponding the inside of the hole can be acquired.

In this way, instead of reading all reports by the report reading button234, setting an imaging condition based on a data set of a selected report and restarting the experiment, similar conditions are searched for based on the current observation purpose and the process leading up to imaging recorded in the report retention unit119, and the imaging condition is set in the image acquisition device103, so that the imaging condition can be set in a short time.

In addition, for example, a report matching with the observation purpose may be extracted by intentionally describing information such as a key word useful for future search in the title description area301or a comment in the report REP inFIG.3.

<Main Effects According to Embodiment>

According to the embodiment, the instruction analysis unit113generates the conversation document in which the conversation uttered by the user is converted into character string data and recognizes, based on the conversation document, the conversation intention of the user including the instruction issued to the image acquisition device. Then, the history retention unit115records, as history information, the conversation document, the conversation intention, and the response of the image acquisition device103to the instruction to the image acquisition device103. Then, the difference analysis unit116divides the report creation period using a timing when the user issues, to the image acquisition device103, an instruction including an intention to save a captured image as a boundary, compares an imaging condition at a timing when an instruction including an intention to save a current captured image is issued with an imaging condition at a timing when an instruction including an intention to save a previous captured image is issued, and extracts a common condition and a differential condition of imaging conditions between a previous report creation period and a current report creation period. Then, the difference analysis unit116outputs the report creation information in which history information divided for each of the report creation periods, and a captured image and a differential condition corresponding to the history information are associated with each other. Then, the report retention unit119creates a report for each of the report creation periods based on the report creation information, and records the created report.

According to this configuration, each report includes contents of the conversation, the captured image, the imaging condition, and the like. Accordingly, an experiment can be easily reproduced by referring to the report, and validity of an observation image can be easily ensured. In addition, when the validity of the observation image is inquired, since an operation leading up to imaging, a conversation leading up to that point, and the like are recorded, it is possible to not only reproduce the experiment but also provide a clear explanation and easily demonstrate the validity of the observation image.

In addition, according to the embodiment, by using the first deep learning model trained by the pair of the learning conversation and the learning document that is a document of the learning conversation, the instruction analysis unit113generates the conversation document based on the conversation uttered by the user. According to this configuration, by using the deep learning model, the conversation uttered by the user can be understood.

In addition, according to the embodiment, the first deep learning model is a long-short term memory. According to this configuration, an existing technique can be suitably applied to the charged particle beam apparatus according to the embodiment.

In addition, according to the embodiment, by using the second deep learning model trained by the pair of the learning document and the learning intention that is a conversation intention corresponding to the learning document, the instruction analysis unit113recognizes the conversation intention of the user based on the conversation document generated by the first deep learning model. According to this configuration, by using the deep learning model, the conversation intention in the conversation uttered by the user can be recognized.

In addition, according to the embodiment, the second deep learning model is a recurrent neural network. According to this configuration, an existing technique can be suitably applied to the charged particle beam apparatus according to the embodiment.

In addition, according to the embodiment, a plurality of input and output devices are provided, a plurality of users use different ones of the input and output devices, and conversations uttered by all the users participating in the experiment are displayed on the input and output devices of the respective users. According to this configuration, the plurality of users can share contents of the experiment even when the users are located at remote locations. In addition, the experiment can be conducted efficiently.

In addition, according to the embodiment, the control unit114reads the report recorded in the report retention unit119and sets, in the image acquisition device103, an imaging condition corresponding to the read report. According to this configuration, it is possible to easily reproduce an experiment.

Second Embodiment

Next, a second embodiment will be described.FIG.5shows a configuration example of a charged particle beam apparatus according to the second embodiment of the invention. As shown inFIG.5, in the embodiment, a plurality of users use one input and output device. An input and output device510inFIG.5has a configuration in which a speaker recognition unit501is added to the input and output device110inFIG.1.

The speaker recognition unit501is connected to the microphone112and recognizes a speaker who inputs a conversation from the microphone112to the input and output device510. The speaker recognition unit501recognizes (identifies and authenticates) the speaker who utters the conversation by speaker recognition, and outputs account information502of the recognized speaker to the history retention unit115.

When performing the speaker recognition, the speaker recognition unit501reads learning information necessary for the speaker recognition at the time of logging in by the log-in and log-out button230(FIG.2). The speaker recognition unit501learns a feature in voice of the user in advance by using the read information. Alternatively, the speaker recognition unit501may use a deep learning model trained in advance by a voice signal received from the microphone112.

The speaker recognition unit501refers to correspondence information in which the user (speaker) is associated with the account information502and acquires the account information502of the user identified by the speaker recognition. Then, the speaker recognition unit501outputs the acquired account information502to the history retention unit115. In this case, even when a plurality of users share one microphone, it is possible to identify the speaker.

The history retention unit115records a conversation document and a conversation intention received from the instruction analysis unit113in association with the account information502received from the speaker recognition unit501. The correspondence information in which the user (speaker) is associated with the account information502is recorded in a storage device in the input and output device501.

Meanwhile, in the embodiment, a microphone may be provided for each user. For example, one microphone112is assigned to one user ID. In this case, at the time of logging in by the log-in and log-out button230inFIG.2, device information for identifying the microphone112is assigned to each user. Accordingly, the speaker recognition unit501can recognize the speaker by detecting the microphone to which the conversation is input. In addition, the speaker recognition unit501may perform speaker recognition on input from each microphone. Accordingly, even when the user inputs a conversation from a microphone different from the microphone registered in advance, the speaker can be accurately identified.

According to the embodiment, a plurality of users use one input and output device110. The input and output device110includes the speaker recognition unit501which recognizes the user who utters the conversation. According to this configuration, since necessary hardware can be shared, space and cost required for the input and output device can be reduced.

In addition, according to the embodiment, since a plurality of input and output devices are not used, a configuration for interlocking the input and output devices is unnecessary.

Third Embodiment

Next, a third embodiment will be described.FIG.6shows a configuration example of a charged particle beam apparatus according to the third embodiment of the invention. An input and output device610inFIG.6has a configuration in which a routine extraction unit601is added to the input and output device110inFIG.1.

The routine extraction unit601reads a plurality of reports from the report retention unit119, and extracts, from the plurality of read reports, an operation commonly performed in a plurality of experiments.

When the routine extraction unit601extracts the common operation from the plurality of reports, the routine extraction unit601assigns a routine key word602to the extracted operation and registers the routine key word602in the instruction analysis unit113. In addition, the routine extraction unit601generates, as a routine command603, a command in the operation corresponding to the routine key word602. Further, the routine extraction unit601outputs the routine key word602and the routine command603to the control unit114. The control unit114associates the input routine key word602and the corresponding routine command603with each other, and registers the input routine key word602and the corresponding routine command603as one set in the storage device.

The number of routine commands603corresponding to one routine key word is not limited to one. A command set including a plurality of commands may correspond to one routine key word.

When the registered routine key word602is input by the user, the instruction analysis unit113outputs the input routine key word602to the control unit114and instructs the control unit114to execute the routine command603.

The control unit114searches for the routine command603corresponding to the routine key word602by using the routine key word602received from the instruction analysis unit113. The control unit114executes the routine command matching with a search condition and controls the image acquisition device103.

For example, the user always performs automatic optical axis adjustment and automatic focus adjustment before an experiment, and observes a sample by shortening a working distance (WD) depending on the sample. In such a case, the routine extraction unit601extracts, for example, the routine key word602“as usual” from a plurality of reports, and generates, as the routine command603with respect to the extracted routine key word602, a command set at least including commands of automatic optical axis adjustment and automatic focus adjustment.

The image acquisition device103executes automatic optical axis adjustment and automatic focus adjustment when the user inputs “as usual” by voice before an experiment is started.

According to the embodiment, when the routine extraction unit601extracts the common operation from the plurality of reports, the routine extraction unit601assigns the routine key word602to the extracted operation and registers the routine key word602in the instruction analysis unit113. Then, the routine extraction unit601generates, as the routine command603, a command in the extracted operation. According to this configuration, it is possible to prevent occurrence of a human error during the experiment and to reduce complexity of tasks by extracting and executing tasks that the user routinely performs.

Fourth Embodiment

Next, a fourth embodiment will be described.FIG.7shows a configuration example of a charged particle beam apparatus according to the fourth embodiment of the invention. An input and output device710inFIG.7has a configuration in which a camera701and an emotion analysis unit702are added to the input and output device110inFIG.1whereas the history retention unit115and the difference analysis unit are deleted.

It is generally known that deep learning is effective in an emotion estimation technique for estimating emotion based on data naturally expressed by a person, such as a facial expression, a gesture, eye contact, and voice.

The camera701acquires a facial expression image705including a facial expression of the first user101and outputs the acquired facial expression image705to the emotion detection unit702. In order to detect emotion also from voice of the first user101, output of the microphone112is input to the emotion detection unit702. By using a deep learning model trained by features in advance, the emotion detection unit702detects the emotion of the first user101based on at least one of the facial expression image705received from the camera701and information on the voice of the first user101input via the microphone112.

For example, the emotion detection unit702detects the emotion of the user based on the facial expression image705and a conversation uttered by the user. For example, when emotion such as “joy” or “surprise” of the user is detected, the emotion detection unit702outputs a signal corresponding to the emotion to the instruction analysis unit113and the control unit114. The emotion detection unit702outputs voice704to the instruction analysis unit113and outputs an imaging instruction703to the control unit114. In addition, the emotion detection unit702outputs the input facial expression image705to the report retention unit119.

The instruction analysis unit113extracts an impression706of the user based on the input voice704, and outputs the extracted impression706to the report retention unit119. In addition, the instruction analysis unit113may generate a conversation document and output the conversation document to the report retention unit119as in the above embodiments.

The report retention unit119creates a report in which the facial expression image705received from the emotion detection unit702, the impression706received from the instruction analysis unit113, the captured image117and the imaging condition118received from the control unit114are associated with each other. The report is recorded in the report retention unit119in a viewable format.

The report in the embodiment includes, for example, an observation image observed when the user feels impressed and a state (facial expression and impression) of the user at that time. Specifically, in the report of the embodiment, it is assumed that observation of a sample using the charged particle beam apparatus is a theme to be freely studied by children. It is assumed that the report at this time requires an experimental result and an impression. When a teacher in a school, a parent, or the like is assumed to be a reader of the report, it is conceivable that the reader is interested in what kind of impression the children have when viewing what observation image, or a state of the children during the experiment.

According to the embodiment, for example, it is possible to automatically record, as the report, an observation image in which the children show interest, uttered contents such as impressions, and a facial expression image showing a state at the time of the experiment. In addition, the report in the embodiment can be used as a material for notifying the teacher or the parent of a result of the free study of the children, or as a preparation material for a report to be created by the children.

Although the invention invented by the inventors has been specifically described above based on the embodiments, the invention is not limited to the embodiments, and various modifications can be made without departing from the gist of the invention. For example, the above embodiments have been described in detail to facilitate understanding of the invention, and the invention is not necessarily limited to those including all the configurations described above. In addition, a part of a configuration of one embodiment can be replaced with a configuration of another embodiment, and a configuration of another embodiment can be added to the configuration of the one embodiment. In addition, a part of a configuration of each embodiment may be added to, deleted from, or replaced with another configuration.

All of these are within the scope of the invention, and numerical values, messages, and the like appearing in sentences and drawings are merely examples, and the effects of the invention will not be impaired when different ones are used.

REFERENCE SIGNS LIST

100: charged particle beam apparatus103: image acquisition device110,130,510,610,710: input and output device113: instruction analysis unit114: control unit115: history retention unit116: difference analysis unit119: report retention unit501: speaker recognition unit601: routine extraction unit701: camera702: emotion detection unit