DEVICE MANAGEMENT SYSTEM, INFORMATION PROCESSING SYSTEM, INFORMATION PROCESSING DEVICE, DEVICE MANAGEMENT METHOD, AND NON-TRANSITORY RECORDING MEDIUM

A device management system includes a first device including first circuitry to output a device identifier of the first device, a second device including second circuitry to acquire the device identifier output by the first device and transmit the device identifier to a communication terminal that communicates with an information processing server, and the information processing server. The information processing server includes third circuitry to receive the device identifier from the communication terminal; and enable the first device to be used in a communication with the communication terminal to process information relating to the communication in response to receiving the device identifier.

CROSS REFERENCE TO RELATED APPLICATIONS

This patent application is based on and claims priority pursuant to 35 U.S.C. §119(a) to Japanese Patent Application No. 2022-026944, filed on Feb. 24, 2022, in the Japan Patent Office, the entire disclosure of which is hereby incorporated by reference herein.

BACKGROUND

Technical Field

The present disclosure relates to a device management system, an information processing system, an information processing device, a device management method, and a non-transitory recording medium.

Related Art

Known telecommunication systems transmit images and audio from one site to one or more other sites in real time to allow users at remote sites to hold a teleconference.

In such telecommunication, a device such as an electronic whiteboard is sometimes used.

There is known a technique for facilitating authentication of participants in a conference. For example, a system of the related art includes an image capturing device that captures an image of the surroundings of the image capturing device and generates moving image (video) of the surroundings. The image capturing device reads a participation certificate, analyzes the participation certificate converted into image data, and accepts participation in a conference.

SUMMARY

In one aspect, a device management system includes a first device including first circuitry to output a device identifier of the first device, a second device including second circuitry to acquire the device identifier output by the first device and transmit the device identifier to a communication terminal that communicates with an information processing server, and the information processing server. The information processing server includes third circuitry to receive the device identifier from the communication terminal; and enable the first device to be used in a communication with the communication terminal to process information relating to the communication, in response to receiving the device identifier.

In another aspect, an information processing system includes circuitry configured to receive, from a communication terminal, a device identifier identifying a first device and being output by the first device and acquired by a second device that communicates with the communication terminal. In response to receiving the device identifier, the circuitry enables the first device to be used in a communication for processing information relating to the communication.

In another aspect, a device management method performed by an information processing system, the method includes receiving, a device identifier identifying a first device from a communication terminal, and enabling the first device to be used in a communication for processing information relating to the communication in response to receiving the device identifier. The device identifier is output by the first device and acquired by a second device that communicates with the communication terminal.

In another aspect, a non-transitory recording medium stores a plurality of program codes which, when executed by one or more processors, causes the processors to perform the method described above.

In another aspect, an information processing device includes circuitry configured to acquire a device identifier output from another information processing device to be used in a communication. In response to receiving the device identifier, the circuitry transmits the device identifier to an information processing system that enables the another information processing device to be used in the communication.

In another aspect, an information processing device includes circuitry configured to acquire a device identifier output from another information processing device to be used in a communication; and transmit the device identifier to a communication terminal. The communication terminal transmits the device identifier to an information processing system, and the information processing system enables the another information processing device to be used in the communication in response to receiving the device identifier.

DETAILED DESCRIPTION

Hereinafter, descriptions are given of an information processing system and a method for managing devices performed by the information processing system as an exemplary embodiment of the present disclosure.

Example of Method of Creating Minutes of Teleconference

An overview of a method of creating minutes using a panoramic image and a screen of an application will be described with reference toFIG.1.FIG.1is a diagram illustrating an overview of creation of record for storing a screen of an application executed during a teleconference, together with a panoramic image of surroundings. As illustrated inFIG.1, a user107at a first site102uses a teleconference service system90to have a teleconference with a user at a second site101.

A record creation system100according to the present embodiment creates a record (minutes) using a horizontal panoramic image (hereinafter “panoramic image”) and a screen provided by an application that executes on a communication terminal10. The panoramic image is captured by a meeting device60that includes an image-capturing device, a microphone, and a speaker. The record creation system100combines audio data received by a teleconference application42and audio data obtained by the meeting device60together and includes the resultant audio data in the record. The overview will be described below.

On the communication terminal10, an information recording application41described below and the teleconference application42are operating. Another application such as a document display application may also be operating. The information recording application41transmits audio data output by the communication terminal10(including audio data received by the teleconference application42from the second site101) to the meeting device60. The meeting device60mixes (combines) audio data obtained by the meeting device60and the audio data received by the teleconference application42together.

(2) The meeting device60includes the microphone. Based on a direction from which the microphone receives sound, the meeting device60performs clipping of a portion including a person speaking (i.e., a talker) from the panoramic image to create a talker image. The meeting device60transmits both the panoramic image and the talker image to the communication terminal10.

The information recording application41operating on the communication terminal10displays a panoramic image203and talker images204. The information recording application41combines the panoramic image203and the talker images204with a screen of a desired application (for example, a screen103of the teleconference application42) selected by the user107. For example, the information recording application41combines the panoramic image203and the talker images204with the screen103of the teleconference application42to create a combined image105such that the panoramic image203and the talker image204are arranged on the left side and the screen103of the teleconference application42is arranged on the right side. Since the processing (3) is repeatedly performed, the resultant combined images105become a moving image (hereinafter, referred to as a combined video). The information recording application41attaches the combined audio data to the combined video to create a video with sound.

In the present embodiment, an example of combining the panoramic image203, the talker images204, and the screen103of the teleconference application42together is described. Alternatively, the panoramic image203. the talker images204, and the screen103of the teleconference application42may be stored separately and arranged on a screen at the time of playback by the information recording application41.

The information recording application41receives an editing operation (performed by the user107to cut off a portion not to be used), and completes the combined video. The combined video is a part of the record.

The information recording application41transmits the created combined video (with sound) to a storage service system70for storage.

The information recording application41extracts the audio data from the combined video (or may keep the original audio data to be attached) and transmits the extracted audio data to an information processing system50. The information processing system50receives the audio data and transmits the audio data to a speech recognition service system80that converts the audio data into text data. The speech recognition service system80converts the audio data into text data. The text data includes data indicating a time, from the start of recording, when a speaker made an utterance.

In the case of real-time conversion into text data, the meeting device60transmits the audio data directly to the information processing system50. The information processing system50transmits the text data obtained by speech recognition to the information recording application41in real time.

(7) The information processing system50additionally stores the text data in the storage service system70storing the combined video. The text data is a part of the record.

The information processing system50performs a charging process for a user according to a service that is used. For example, the charge is calculated based on an amount of the text data, a file size of the combined video, a processing time, or the like.

As described above, the combined video displays the panoramic image203of the surroundings including the user107and the talker images204as well as the screen of the application such as the teleconference application42displayed in the teleconference. When a participant or someone who has not attended the teleconference views the combined video as the minutes of the teleconference, the teleconference is reproduced with the realism.

Overview of Process

Next, association processing between the electronic whiteboard2(an example of first device and an example of another information processing device) and the meeting device60(an example of second device and an example of information processing device) will be described with reference toFIG.2. The information processing system50serves as an information processing server. The information processing system50associates electronic whiteboard2with the conference to enable the electronic whiteboard2to be used in a teleconference for processing information related to the teleconference (e.g., storing objects handwritten on the electronic whiteboard2in association with the teleconference). Such associating is performed by a user in a conventional system.

FIG.2is a diagram illustrating an overview of an operation performed by the information processing system50for associating the electronic whiteboard2with the meeting device60.(1) A participant brings the meeting device60to a conference room in which the electronic whiteboard2(an example of the first device and an example of another device) is installed and places the meeting device60on a desk. The meeting device60is placed near the electronic whiteboard2.(2) The meeting device60captures an image of a two-dimensional code in which a device identifier (ID) of the electronic whiteboard2is embedded, and transmits image data thereof to the communication terminal10.(3) The communication terminal10detects the device ID from the image data and transmits the device ID to the information processing system50.(4) In response to receiving the device ID, the information processing system50issues a conference identifier (ID) and associates the device ID of the electronic whiteboard2with the conference ID.The timing at which the information processing system50issues the conference ID is not limited thereto. For example, the conference ID may be issued in advance by an operation of the user, or may be automatically issued at the conference start time in cooperation with a scheduler.(5) The information processing system50transmits a registration completion notification including the conference ID to the electronic whiteboard2and the communication terminal10. As the electronic whiteboard2and the communication terminal10specify the conference ID and communicate with the information processing system50, data handled by the electronic whiteboard2and the communication terminal10can be associated with the same conference. The information processing system50stores the image data (the panoramic image and the talker image) and the audio data acquired by the meeting device60, the hand-drafted data input to the electronic whiteboard2, and the like in association with the conference ID.

As described above, in the record creation system100according to the present embodiment, the meeting device60acquires the device ID of the device used in the conference room, and the communication terminal10transmits the device ID to the information processing system50. Then, the information processing system50associates the device with the information processing system50as the device used in the conference. Since the meeting device60is connected to the communication terminal10that communicates with the information processing system50, the meeting device60is also associated with the conference.

This configuration obviates the trouble of the user of capturing the two-dimensional code displayed by the device with the camera or registering the device and the meeting device60in the information processing system50for a conference. With this configuration, a plurality of devices (the meeting device60and the electronic whiteboard2) can be associated with the conference (made usable) with minimum user intervention. Examples of enabling a device usable include, but not limited to: making the device usable in a conference: enabling the electronic whiteboard2to transmit hand-drafted stroke data, an image, or the like to the information processing system50; enabling creation of minutes using information input to the electronic whiteboard2; associating the electronic whiteboard2with the conference; enabling the electronic whiteboard2and the information processing system50to transmit and receive data to and from each other: causing the electronic whiteboard2to participate in a cloud electronic whiteboard service (a service for multiple communication terminals to write or draw a stroke or the like on the same screen in a teleconference). In addition, when an electronic whiteboard is enabled to be usable in a conference, the electronic whiteboard can be prevented from participating in another conference held at the same time. In addition, when an electronic whiteboard is made usable in a conference information displayed thereon and information (stroke information) drawn thereon can be included in minutes of the conference created by the information processing system or the like.

Terminology

The term “application (app)” refers to software developed or used for a specific function or purpose, not software for operating a computer itself. That is, “application” is not an operating system (OS). Types of such applications include a native application and a web application.

The expression “application being executed” refers to an application in a state from the activation of an application to the end of the application. An application is not necessarily active (an application in the foreground) and may operate in the background.

An “image of the surroundings acquired by the meeting device 60” refers to an image captured in a wider angle of view than a normal angle of view in the horizontal direction. In the present embodiment, the image of the surroundings is referred to as a “panoramic image.” A panoramic image is an image having an angle of view of 180 degrees to 360 degrees in substantially the horizontal direction. The panoramic image is not necessarily captured by a single meeting device, and may be captured by a combination of a plurality of image-capturing devices having an ordinary angle of view.

The term “record” refers to information recorded (recorded information) by the information recording application41. When the information recording application41records the screen of the teleconference application42, the record may serve as minutes of a teleconference. The “record” includes, for example, a combined video (with sound) and text data obtained by performing speech recognition on the sound.

The term “tenant” refers to a group of users (such as a company, a local government, or an organization that is a part of such a company or local government) that has a contract to receive a service from a service provider. In the present embodiment, assuming that the tenant has a contract with the service provider, creation of the record and conversion into text data are performed.

The term “telecommunication” refers to audio-and-video-based communication with a counterpart at a physically remote site, using software and communication terminals.

A remote conference (teleconference) and a seminar are examples of telecommunication. A conference may also be referred to as an assembly, a meeting, an arrangement, a gathering, a meet, or a meet-up. A seminar may also be referred to as a workshop, a study meeting, a study session, or a training session.

The term “site” refers to a place where an activity is performed. A conference room is an example of the site. The conference room is a room installed for use in a conference.

The term “sound” and “audio” refers to an utterance made by a person, a surrounding sound, or the like. The term “audio data” refers to data to which the audio is converted. However, in the present embodiment, the audio and the audio data will be described without being strictly distinguished from each other.

The “first device” may be any device that display information. In the present embodiment, the first device is described with the term “electronic whiteboard.” The electronic whiteboard may also be referred to as an electronic information board or the like. A projector is known as the equivalent to an electronic whiteboard. Alternatively, the first device may be a digital signage, a television, a display, a multifunction peripheral, a video conference terminal, or the like in other embodiments.

The term “information related to communication” refers to information recorded in communication such as a conference, and is, for example, information displayed by an electronic whiteboard, an image captured by an image-capturing device, or voice of a speaker. Examples of devices that process information related to communication include the electronic whiteboard2and the meeting device60. The meeting device60may include an image-capturing device.

To the first device and the communication terminal, the same identification information (information for associating a plurality of devices in communication) is transmitted. The same identification information is “conference ID” in the present embodiment, but may be any information.

Example of System Configuration

A system configuration of the record creation system100will be described with reference toFIG.3.FIG.3illustrates an example of the configuration of the record creation system100.FIG.3illustrates one site (the first site102on which the meeting device60is located) among a plurality of sites between which a teleconference is held. The communication terminal10at the first site102communicates with the information processing system50, the storage service system70, and the teleconference service system90via a network. The meeting device60and the electronic whiteboard2are disposed at the first site102. The communication terminal10is connected to the meeting device60via, for example, a Universal Serial Bus (USB) cable to communicate therewith. The meeting device60, the electronic whiteboard2, and the information processing system50operate as a device management system.

At least the information recording application41and the teleconference application42operate on the communication terminal10. The teleconference application42can communicate with the communication terminal10at the second site101via the teleconference service system90that resides on the network to allow users at the remote sites to participate in a teleconference. The information recording application41uses functions of the information processing system50and the meeting device60to create the record of the teleconference hosted by the teleconference application42.

In the present embodiment, a description is given of an example in which the record of a teleconference is created. However, in another example, the conference is not necessarily held among remote sites. That is, aspects of the present disclosure are applicable to a conference held among the participants present at one site. In this case, the image captured by the meeting device60and the audio received by the meeting device60are independently stored without being combined. The rest of the processing performed by the information recording application41is similar to that of the present embodiment.

The communication terminal10includes a built-in (or external) camera having an ordinary angle of view. The camera of the communication terminal10captures an image of a front space including the user107who operates the communication terminal10. Images captured by the camera having an ordinary angle of view are not panoramic images. In the present embodiment, the built-in camera having the ordinary angle of view primarily captures planar images that are not curved like spherical images. Thus, the user can participate in a teleconference using the teleconference application42as usual without paying attention to the information recording application41. The information recording application41and the meeting device60do not affect the teleconference application42except for an increase in the processing load of the communication terminal10. The teleconference application42can transmit a panoramic image or a talker image captured by the meeting device60to the teleconference service system90.

The information recording application41communicates with the meeting device60to create a record of a conference. The information recording application41also synthesizes audio received by the meeting device60and audio received by the teleconference application42from another site. The meeting device60is a device for a meeting, including an image-capturing device that captures a panoramic image, a microphone, and a speaker. The camera of the communication terminal10can capture an image of only a limited range of the front space. In contrast, the meeting device60can capture an image of the entire surroundings (not necessarily the entire surroundings) around the meeting device60. The meeting device60can always keep a plurality of participants106illustrated inFIG.3within the angle of view.

In addition, the meeting device60cuts out a talker image from a panoramic image. The meeting device60is placed on a table inFIG.3, but may be placed anywhere in the first site102. Since the meeting device60can capture a spherical image, the meeting device60may be disposed on a ceiling, for example.

The information recording application41displays a list of applications executing on the communication terminal10, combines images for the above-described record (creates the combined video), plays the combined video, receives editing, and the like. Further, the information recording application41displays a list of teleconferences already held or are to be held in the future. The list of teleconferences is used in information on the record to allow the user to link a teleconference with the record.

The teleconference application42establishes communication connection with the second site101, transmits and receives images and sound to and from the second site101, displays images, and outputs audio.

The information recording application41and the teleconference application42each may be a web application or a native application. A web application is an application in which a program on a web server cooperates with a program on a web browser to perform processing, and is not to be installed on the communication terminal10. A native application is an application that is installed and used on the communication terminal10. In the present embodiment, both the information recording application41and the teleconference application42are described as native applications.

The communication terminal10may be a general-purpose information processing apparatus having a communication function, such as a personal computer (PC), a smartphone, or a tablet terminal, for example. Alternatively, the communication terminal10is, for example, an electronic whiteboard, a game console, a personal digital assistant (PDA), a wearable PC, a car navigation system, an industrial machine, a medical device, or a networked home appliance. The communication terminal10may be any apparatus on which the information recording application41and the teleconference application42operate.

The electronic whiteboard2displays, on a display, data handwritten on a touch panel with an input device such as a pen or a finger. The electronic whiteboard2can communicate with the communication terminal10or the like in a wired or wireless manner, and capture a screen displayed by the communication terminal10and display the screen on the display. The electronic whiteboard2can convert hand-drafted data into text data, and share information displayed on the display with the electronic whiteboard2at another site. The electronic whiteboard2may be a whiteboard, not including a touch panel, onto which a projector projects an image. The electronic whiteboard2may be a tablet terminal, a laptop computer or PC, a PDA, a game console, or the like including a touch panel.

The electronic whiteboard2can communicate with the information processing system50. For example, after being powered on, the electronic whiteboard2performs polling on the information processing system50to receive information from the information processing system50.

The information processing system50is implemented by one or more information processing apparatuses deployed over a network. The information processing system50includes one or more server applications that perform processing in cooperation with the information recording application41, and an infrastructure service. The server applications manage, for example, a list of teleconferences, records of teleconferences, and various settings and storage paths.

The infrastructure service performs user authentication, makes a contract, performs charging processing, and the like.

All or some of the functions of the information processing system50may reside in a cloud environment or in an on-premises environment. The information processing system50may be implemented by a plurality of server apparatuses or a single information processing apparatus. For example, the server applications and the infrastructure service may be provided by separate information processing apparatuses. Further, each function of the server applications may be provided by an individual information processing apparatus. The information processing system50may be integral with the storage service system70and the speech recognition service system80described below.

The storage service system70is a storage means on a network, and provides a storage service for accepting the storage of files and the like. Examples of the storage service system70include MICROSOFT ONEDRIVE, GOOGLE WORKSPACE, and DROPBOX. The storage service system70may be on-premises network-attached storage (NAS) or the like.

The speech recognition service system80provides a service of speech recognition on audio data and converting the audio data into text data. The speech recognition service system80may be a general-purpose commercial service or a part of the functions of the information processing system50.

Example of Hardware Configuration

A hardware configuration of the information processing system50and the communication terminal10according to the present embodiment will be described with reference toFIG.4.

Information Processing System and Communication Terminal

FIG.4is a diagram illustrating an example of a hardware configuration of the information processing system50and the communication terminal10according to the present embodiment. As illustrated inFIG.4, the information processing system50and the communication terminal10each are implemented by a computer and each include a central processing unit (CPU)501, a read-only memory (ROM)502. a random access memory (RAM)503, a hard disk (HD)504, a hard disk drive (HDD) controller505, a display506, an external device interface (I/F)508, a network I/F509, a bus line510, a keyboard511. a pointing device512, an optical drive514, and a medium I/F516.

The CPU501controls the entire operations of the information processing system50and the communication terminal10. The ROM502stores programs such as an initial program loader (IPL) to boot the CPU501. The RAM503is used as a work area for the CPU501. The HD504stores various kinds of data such as a program. The HDD controller505controls reading or writing of various kinds of data from or to the HD504under control of the CPU501. The display506displays various kinds of information such as a cursor, a menu, a window, characters, or an image. The external device I/F508is an interface for connecting various external devices. Examples of the external devices in this case include, but are not limited to, a USB memory and a printer. The network I/F509is an interface for performing data communication via a network. The bus line510is, for example, an address bus or a data bus for electrically connecting the components such as the CPU501illustrated inFIG.4to one another.

The keyboard511is a kind of an input device including a plurality of keys used for inputting characters, numerical values, various instructions, or the like. The pointing device512is a kind of an input device used to select or execute various instructions, select a target for processing, or move a cursor. The optical drive514controls the reading or writing of various kinds of data from or to an optical recording medium513that is an example of a removable recording medium. The optical recording medium513may be a compact disc (CD), a digital versatile disc (DVD), a BLU-RAY disc, or the like. The medium I/F516controls reading or writing (storing) of data from or to a recording medium515such as a flash memory.

Meeting Device

A hardware configuration of the meeting device60will be described with reference toFIG.5.FIG.5is a block diagram illustrating an example of a hardware configuration of the meeting device60that can create a 360-degree video of surroundings according to the present embodiment. In the following description, the meeting device60is assumed to be a device that uses an imaging element to capture a 360-degree image of the surroundings of the meeting device60at a predetermined height, to produce a video. The number of imaging elements may be one or two or more. The meeting device60is not necessarily a dedicated device and may be a PC, a digital camera, a smartphone, or the like to which an imaging unit for a 360-degree video is externally attached so as to implement substantially the same functions as the meeting device60.

As illustrated inFIG.5, the meeting device60includes an imaging unit601, an image processing unit604. an image capture control unit605, microphones608a,608b. and608c(collectively “microphones 608”), an audio processing unit609, a CPU611, a ROM612, a static random access memory (SRAM)613, a dynamic random access memory (DRAM)614, an operation device615, an external device I/F616, a communication unit617, an antenna617a, and an audio sensor618. The external device I/F616includes a socket terminal for Micro-USB.

The imaging unit601includes a wide-angle lens602(so-called fisheye lens) having an angle of view of 360 degrees to form a hemispherical image, and an imaging element603(image sensor) provided for the wide-angle lens602. The imaging element603includes an image sensor such as a complementary metal oxide semiconductor (CMOS) sensor or a charge coupled device (CCD) sensor, a timing generation circuit, and a group of registers. The image sensor converts an optical image formed by the wide-angle lens602into an electric signal to output image data. The timing generation circuit generates horizontal or vertical synchronization signals, pixel clocks, and the like for the image sensor. Various commands, parameters, and the like for operations of the imaging element are set in the group of registers.

The imaging element603(image sensor) of the imaging unit601is connected to the image processing unit604via a parallel I/F bus. On the other hand, the imaging element603of the imaging unit601is connected to the image capture control unit605via a serial I/F bus such as an inter-integrated circuit (I2C) bus. The image processing unit604, the image capture control unit605, and the audio processing unit609, each of which may be implemented by a circuit, are connected to the CPU611via a bus610. The ROM612, the SRAM613, the DRAM614, the operation device615, the external device I/F616, the communication unit617, the sound sensor618, and the like are also connected to the bus610.

The image processing unit604obtains image data output from the imaging element603through the parallel I/F bus and performs predetermined processing on the image data to create data of a panoramic image and data of a talker image from a fisheye image. The image processing unit604combines the panoramic image and the talker image or the like together to output a single video (moving image).

The image capture control unit605usually serves as a master device, whereas the imaging element603usually serves as a slave device. The image capture control unit605sets commands and the like in the groups of registers of the imaging element603through the 12C bus. The image capture control unit605receives the commands and the like from the CPU611. The image capture control unit605obtains status data and the like in the groups of registers of the imaging element603through the I2C bus. The image capture control unit605then sends the obtained data to the CPU611.

The image capture control unit605instructs the imaging element603to output image data at a timing when an image-capturing start button of the operation device615is pressed or a timing when the image capture control unit605receives an image-capturing start instruction from the CPU611. In some cases, the meeting device60supports a preview display function and a video display function of a display (e.g.. a display of a PC or a smartphone). In this case, the image data is consecutively output from the imaging elements603at a predetermined frame rate (frames per minute).

When the meeting device60includes a plurality of imaging elements603, the image capture control unit605operates in cooperation with the CPU611to synchronize the output timing of image data from the plurality of imaging elements603. In the present embodiment, the meeting device60does not include a display. However, in some embodiments, the meeting device60includes a display.

The microphones608convert sound into audio (signal) data. The audio processing unit609receives the audio data output from the microphones608a,608b, and608cvia an I/F bus, mixes (combines) the audio data, and performs predetermined processing on the audio data. The audio processing unit609also determines a direction of an audio source (talker) from a level of the audio (volume) input from the microphones608ato608c.

The CPU611controls the entire operations of the meeting device60and performs desirable processing. The ROM612stores various programs for operating the meeting device60. Each of the SRAM613and the DRAM614is a work memory and stores programs being executed by the CPU611or data being processed. In particular, in one example, the DRAM614stores image data being processed by the image processing unit604and processed data of an equirectangular projection image.

The operation device615collectively refers to various operation buttons such as an image-capturing start button. The user operates the operation device615to start image-capturing or recording, power on or off the meeting device60, establish a connection, perform communication, and input settings such as various image-capturing modes and image-capturing conditions.

The external device I/F616is an interface for connecting various external devices. The extemal device in this case is, for example, a personal computer (PC). The video data or still image data stored in the DRAM614is transmitted to an external communication terminal or stored in an external medium via the external device I/F616.

The communication unit617is implemented by, for example, a network interface circuit. The communication unit617may communicate with a cloud server via the Internet using a wireless communication technology such as Wireless Fidelity (Wi-Fi) via an antenna617a of the meeting device60and transmit the video data and the image data stored in the DRAM614to the cloud server. Further, the communication unit617may be able to communicate with nearby devices using a short-range wireless communication technology such as BLUETOOTH LOW ENERGY (BLE) or the near field communication (NFC).

The sound sensor618is a sensor that acquires 360-degree audio data in order to identify the direction from which a loud sound is input within a 360-degree space around the meeting device60(on a horizontal plane). The audio processing unit609determines the direction in which the volume of the sound is highest, based on the input 360-degree audio parameter, and outputs the direction from which the sound is input within the 360-degree space.

Note that another sensor (such as an azimuth/accelerometer or a Global Positioning System (GPS)) may calculate an azimuth, a position, an angle, an acceleration, or the like and use the calculated azimuth, position, angle, acceleration, or the like in image correction or position information addition.

The image processing unit604generates a panoramic image in the following method. The CPU611performs predetermined camera image processing such as Bayer interpolation (red green blue (RGB) supplementation processing) on raw data input by an image sensor that inputs a spherical image, to generate a wide-angle image (a video including curved-surface images). Further, the CPU611performs unwrapping processing (distortion correction processing) on the wide-angle image lens (the video including curved-surface images) to generate a panoramic image (a video including planar images) of the surroundings in 360 degrees around the meeting device60.

The CPU611creates a talker image according to a method below. The CPU611generates a talker image on which a talker is cut out from a panoramic image (a video including planar images) of the surroundings in 360 degrees around the meeting device60. The CPU611cuts out, from the panoramic image, a talker image corresponding the direction of the talker which is the input direction of the audio determined from 360 degrees, using the audio sensor618and the audio processing unit609. For cutting out an image of a person based on the input direction of the audio, specifically, the CPU611cuts out a 30-degree portion around the input direction of the audio identified from 360 degrees, and performs face detection on the 30-degree portion to cut out the talker image. The CPU611further identifies talker images of a predetermined number of persons (e.g., three persons) who have most recently spoken, among talker images cut out from the panoramic image.

The panoramic image and one or more talker images may be individually transmitted to the information recording application41. Alternatively, the meeting device60may create one image combined from the panoramic image and the one or more talker images and transmit the one image to the information recording application41. In the present embodiment, the panoramic image and one or more talker images are individually transmitted from the meeting device60to the information recording application41.

FIG.6AandFIG.6Bare diagrams illustrating an image capture range of the meeting device60. As illustrated inFIG.6A,the meeting device60captures an image of a 360-degree range in the horizontal direction. As illustrated inFIG.6B, the meeting device60has an image capture range extending predetermined angles up and down from a 0-degree direction that is horizontal to the height of the meeting device60.

FIG.7is a schematic diagram illustrating a panoramic image and cutting out talker images from the panoramic image. As illustrated inFIG.7, an image captured by the meeting device60is a portion110of a sphere, and thus has a three-dimensional shape. As illustrated inFIG.6B, the meeting device60divides angles of view into the predetermined degrees up and down and by the predetermined angle in the horizontal direction to perform perspective projection conversion on each of the angles of view. A predetermined number of planar images are obtained by performing the perspective projection conversion on the entire 360-degree range in the horizontal direction without gaps. Thus, a panoramic image203is obtained by laterally connecting the predetermined number of planar images. The meeting device60performs face detection on a predetermined range around the sound direction in the panoramic image203, and clips 15-degree leftward and rightward ranges from the center of the face (i.e., a 30-degree range in total) to create a talker image204.

Electronic Whiteboard

FIG.8is a diagram illustrating an example of a hardware configuration of the electronic whiteboard2. As illustrated inFIG.8, the electronic whiteboard2includes a CPU401, a ROM402, a RAM403, a solid state drive (SSD)404, a network I/F405, and an external device I/F406.

The CPU401controls operations of the entire electronic whiteboard2. The ROM402stores a program such as an IPL to boot an operating system (OS). The RAM403is used as a work area for the CPU401. The SSD404stores various kinds of data such as a program for the electronic whiteboard2. The network I/F405controls communication with a communication network. The external device I/F406is an interface for connecting various external devices. Examples of the external devices in this case include, but not limited to, a USB memory430and externally-connected devices such as a microphone440, a speaker450, and a camera460.

The electronic whiteboard2further includes a capture device411, a graphics processing unit (GPU)412, a display controller413, a contact sensor414, a sensor controller415, an electronic pen controller416, a short-range communication circuit419, an antenna419aof the short-range communication circuit419, a power switch422, and a selection switch group423.

The capture device411causes a display of an external PC470to display a still image or a video based on image data captured by the capturing device. The GPU412is a semiconductor chip that exclusively handles graphics. The display controller413controls and manages displaying of a screen to display an image output from the GPU412on a display480. The contact sensor414detects a touch of an electronic pen490, a user’s hand H, or the like onto the display480. The sensor controller415controls processing of the contact sensor414. The contact sensor414receives a touch input and detects coordinates of the touch input according to the infrared blocking system. The inputting and detecting a coordinate may be as follows. For example, two light receiving and emitting devices are disposed at both ends of the upper face of the display480, and a reflector frame surrounds the periphery of the display480. The light receiving and emitting devices emit a plurality of infrared rays in parallel to a surface of the display480. The rays are reflected by the reflector frame, and a light-receiving element receives light returning through the same optical path of the emitted infrared rays. The contact sensor414outputs, to the sensor controller415, position information (a position on the light-receiving elements) of an infrared ray that is emitted from the two light receiving and emitting devices and then blocked by an object. Based on the position information of the infrared ray, the sensor controller415detects specific coordinates of the position touched by the object. The electronic pen controller416communicates with the electronic pen490by BLUETOOTH to detect a touch by the tip or bottom of the electronic pen490to the display480. The short-range communication circuit419is a communication circuit that is compliant with Near Field Communication (NFC), BLUETOOTH, or the like. The power switch422is used for powering on and off the electronic whiteboard2. The selection switch group423is a group of switches for adjusting brightness, hue, etc., of display on the display480.

The electronic whiteboard2further includes a bus line410. The bus line410is, for example, an address bus or a data bus for electrically connecting the components such as the CPU401illustrated inFIG.8to one another.

Note that the contact sensor414is not limited to a touch sensor of the infrared blocking system, and may be a capacitive touch panel that detects a change in capacitance to identify the touched position. The contact sensor414may be a resistive-film touch panel that identifies the touched position based on a change in voltage across two opposing resistive films. The contact sensor414may be an electromagnetic inductive touch panel that detects electromagnetic induction generated by a touch of an object onto a display to identify the touched position. In addition to the devices described above, various types of detection devices may be used as the contact sensor414. The electronic pen controller416may determine whether there is a touch of another part of the electronic pen490such as a part of the electronic pen490held by the user as well as the tip and the bottom of the electronic pen490.

Functions

A description is now given of a functional configuration of the record creation system100, with reference toFIG.9.FIG.9is a block diagram illustrating functional configurations of the communication terminal10, the meeting device60, and the information processing system50of the record creation system100according to the present embodiment.

Communication Terminal

The information recording application41operating on the communication terminal10implements a communication unit11, an operation reception unit12, a display control unit13, an app screen acquisition unit14, an audio reception unit15, a device communication unit16, a recording control unit17. an audio data processing unit18. a replay unit19, an upload unit20. an editing unit21, a code analysis unit22, and a time measuring unit25. These units of functions on the communication terminal10are implemented by or caused to function by one or more of the components illustrated inFIG.4operating in accordance with instructions from the CPU501according to the information recording application41loaded from the HD504to the RAM503. The communication terminal10also includes a storage unit1000implemented by the HD504or the like illustrated inFIG.4. The storage unit1000includes an information storage area1001, which is implemented by a database, for example.

The communication unit11transmits and receives various types of information to and from the information processing system50via a communication network.

For example, the communication unit11receives a list of teleconferences from the information processing system50and transmits an audio data recognition request to the information processing system50.

The display control unit13control display of various screens serving as user interfaces in the information recording application41in accordance with screen transitions set in the information recording application41. The operation reception unit12receives various operations input to the information recording application41.

The app screen acquisition unit14acquires a desktop screen or a screen displayed by an application selected by a user from an operating system (OS) or the like. When the application selected by the user is the teleconference application42. a screen (including e.g., an image of each site and an image of a material or document displayed) generated by the teleconference application42is obtained.

The audio reception unit15acquires audio data received by the communication terminal10from the teleconference application42in a teleconference. Note that the audio data acquired by the audio reception unit15does not include sound collected by the communication terminal10. This is because the meeting device60collects sound.

The device communication unit16communicates with the meeting device60using a USB cable or the like. Alternatively, the device communication unit16may communicate with the meeting device60via a wireless local area network (LAN) or BLUETOOTH. The device communication unit16receives the panoramic image and the talker image from the meeting device60. and transmits the audio data acquired by the audio reception unit15to the meeting device60. The device communication unit16receives the audio data combined by the meeting device60.

The recording control unit17combines the panoramic image and the talker image received by the device communication unit16and the screen of the application acquired by the app screen acquisition unit14together, to create a combined image. The recording control unit17connects the repeatedly created combined images in time series to create a combined video, and attaches the audio data combined by the meeting device60to the combined video, to create a combined video with sound.

The audio data processing unit18requests the information processing system50to convert, into text data, the audio data extracted by the recording control unit17from the combined video with sound or the combined audio data received from the meeting device60.

The replay unit19plays the combined video. The combined video is stored in the communication terminal10during recording, and then uploaded to the information processing system50.

After the teleconference ends, the upload unit20transmits the combined video to the information processing system50.

The editing unit21edits the combined video (e.g., deletes a portion of the combined video or combines a plurality of combined videos) in accordance with a user operation.

The code analysis unit22detects a two-dimensional code included in the panoramic image and analyzes the two-dimensional code to acquire a device ID.

The time measuring unit25measures the time from when the information recording application41is activated to when the two-dimensional code is received from the meeting device60. When a predetermined time has elapsed, the time measuring unit25notifies the display control unit13of the elapse of the predetermined time, and the display control unit13displays an error dialog.

FIG.10illustrates example items of information on the recorded video, stored in the information storage area1001. The information on the recorded video includes items such as “conference ID.” “recording ID,” “update date/time,” “title,” “upload,” and “storage location.” When a user logs into the information processing system50, the information recording application41downloads conference information from a conference information storage area5001of the information processing system50. The conference ID or the like included in the conference information is reflected in the information on the recorded video. The information on the recorded video inFIG.10is stored by the communication terminal10operated by a certain user.

The item “conference ID” is identification information identifying a held teleconference (communication identifier identifying a communication). The conference ID is assigned when a schedule of the teleconference is registered to a conference management system9, or is assigned by the information processing system50in response to a request from the information recording application41.

The item “recording ID” is identification information identifying a combined video recorded during the teleconference.

The recording ID is assigned by the meeting device60, but may be assigned by the information recording application41or the information processing system50. Different recording IDs are assigned to a same conference ID in a case where the recording is suspended in the middle of the teleconference but is started again for some reason.

The item “update date/time” represents the date and time when the combined video is updated (or recording is ended). When the combined video is edited, the update date and time is the date and time of editing.

The item “title” is a name of the conference. The title may be set when the conference is registered to the conference management system9, or may be set by the user in any manner.

The item “uploaded” indicates whether the combined video has been uploaded to the information processing system50.

The item “storage location” indicates a location, such as uniform resource locator (URL) or file path, where the combined video and the text data are stored in the storage service system70. The item “storage location” allows the user to view the uploaded combined video as desired. Note that the combined video and the text data are stored with different file names following the URL, for example.

Meeting Device

Referring back toFIG.9, the description is continued. The meeting device60includes a terminal communication unit61, a panoramic image generation unit62(acquisition unit), a talker image generation unit63, a sound collection unit64, and an audio synthesis unit65. These functional units of the meeting device60are implemented by or caused to function by one or more of the components illustrated inFIG.5operating in accordance with instructions from the CPU611according to the control program loaded from the ROM612to the DRAM614.

The terminal communication unit61communicates with the communication terminal10using a USB cable or the like. The connection of the terminal communication unit61to the communication terminal10is not limited to a wired cable, but includes connection by a wireless LAN, BLUETOOTH, or the like.

The panoramic image generation unit62generates a panoramic image. The talker image generation unit63generates a talker image. The method of generating a panoramic image and a talker image has been described with reference toFIGS.6A to7. The panoramic image generation unit62also serves as an acquisition unit that acquires image data.

The sound collection unit64converts sound received by the microphone of the meeting device60into audio data (digital data). Thus, the utterances (speeches) made by the user and the participants at the site where the communication terminal10is installed are collected.

The audio synthesis unit65combines the audio data transmitted from the communication terminal10and the sound collected by the sound collection unit64. Accordingly, the speeches uttered at the second site101and those uttered at the first site102are combined.

Information Processing System

The information processing system50includes a communication unit51, an authentication unit52, a screen generation unit53, a communication management unit54, a device management unit55, and a text conversion unit56. These functional unit of the information processing system50are implemented by or caused to function by one or more of the components illustrated inFIG.4operating in accordance with instructions from the CPU501according to the control program loaded from the HD504to the RAM503. The information processing system50also includes a storage unit5000implemented by the HD504or the like illustrated inFIG.4. The storage unit5000includes the conference information storage area5001, a record information storage area5002, and an association storage area5003each of which is implemented by a database, for example.

The communication unit51transmits and receives various kinds of information to and from the communication terminal10. For example, the communication unit51transmits a list of teleconferences to the communication terminal10, and receives a request of speech recognition on audio data from the communication terminal10.

The authentication unit52authenticates a user who operates the communication terminal10. For example, the authentication unit52authenticates a user based on whether authentication information (a user ID and a password) included in an authentication request received by the communication unit51matches authentication information held in advance. The authentication information may be a card number of an integrated circuit (IC) card, biometric authentication information of a face, a fingerprint, or the like. The authentication unit52may use an external authentication system or an authentication method such as Open Authorization (OAuth) to perform authentication.

The screen generation unit53generates screen information representing a screen to be displayed with a web application by the communication terminal10. The screen information is described in Hyper Text Markup Language (HTML), Extended Markup Language (XML), Cascade Style Sheet (CSS), or JAVASCRIPT, for example.

The communication management unit54acquires information related to a teleconference from the conference management system9by using an account of each user or a system account assigned to the information processing system50. The communication management unit54stores conference information of a scheduled conference in association with a conference ID in the conference information storage area5001. The communication management unit54acquires conference information for which a user belonging to the tenant has a right to view. Since the conference ID is set for a conference, the teleconference and the record are associated with each other by the conference ID.

In response to receiving device IDs of the electronic whiteboard2and the meeting device60to be used in the conference, the device management unit55stores these device IDs, in association with the teleconference, in the association storage area5003. Accordingly, the conference ID, the device ID of the electronic whiteboard2, and the device ID of the meeting device60are associated with each other. Since the combined video is also associated with the conference ID. the hand-drafted data input on the electronic whiteboard2is also associated with the combined video. In response to the end of recording (the end of the conference), the device management unit55deletes the association from the association storage area5003.

The text conversion unit56uses the external speech recognition service system80to convert, into text data, audio data requested to be converted into text data by the communication terminal10. In some embodiments, the text conversion unit56may perform this conversion.

FIG.11illustrates an example of conference information stored in the conference information storage area5001and managed by the communication management unit54. The communication management unit54uses the aforementioned account to acquire a list of teleconferences for which a user belonging to a tenant has a right to view. In the present embodiment, teleconferences are used as an example. However, the list of teleconferences also includes a conference held in a single conference room.

The conference information is managed with the conference ID, which is associated with the items “participant,” “title,” “start date and time,” “end date and time,” “place,” and the like. These items are an example of the conference information, and the conference information may include other information.

The item “participant” represents participants of the conference.

The item “title” represents a content of the conference such as a name of the conference or an agenda of the conference.

The item “start date and time” indicates a date and time at which the conference is scheduled to be started.

The item “end date and time” indicates a date and time at which the conference is scheduled to end.

The item “place” represents a place where the conference is held such as a name of a conference room, a name of a branch office, or a name of a building.

The item “electronic whiteboard” represents a device ID of the electronic whiteboard2used in the conference.

The item “meeting device” indicates identification information of the meeting device60used in the conference.

As illustrated inFIGS.10and11, a combined video recorded at a conference is identified by the conference ID.

The information on the recorded video stored in the record information storage area5002may be the same as the information illustrated inFIG.10. However, the information processing system50has a list of combined videos recorded by all users belonging to the tenant.

FIG.12illustrates an example of association information associating a conference ID with the device IDs of the electronic whiteboard2and the meeting device60. The association information is stored in the association storage area5003. The association information is held from when the information recording application41transmits the device ID to the information processing system50to when the recording ends.

Electronic Whiteboard

FIG.13is a block diagram illustrating functional configurations of the electronic whiteboard2according to the present embodiment. The electronic whiteboard2includes a contact position detection unit31, a drawing data generation unit32, a data recording unit33, a display control unit34, a code generation unit35(output unit), and a communication unit36. The respective functions of the electronic whiteboard2are functions or means that are implemented by one or more of the components illustrated inFIG.8obeying instructions from the SSD404according to a program loaded to the RAM403from the CPU401.

The contact position detection unit31detects coordinates of a position where the electronic pen490has touched the contact sensor414. The drawing data generation unit32acquires the coordinates of the position touched by the tip of the electronic pen490from the contact position detection unit31. The drawing data generation unit32interpolates a sequence of coordinate points and links the resulting coordinate points to generate stroke data.

The display control unit34displays hand-drafted data, a menu to be operated by the user, and the like on the display.

The data recording unit33stores, in an object information storage area3002, information on hand-drafted data hand-drawn on the electronic whiteboard2, a graphic such as a circle or triangle, a stamp of “DONE” or the like, a PC screen, and a file. Each of the hand-drafted data, the graphic, the image such as a PC screen, and the file is treated as an object. Regarding handwritten data, a set of stroke data grouped is stored as one object. Grouping is made by time due to interruption of input of handwriting or by the position where the handwriting is input.

The communication unit36is connected to Wi-Fi or a LAN and communicates with the information processing system50. The communication unit36transmits object information to the information processing system50. receives object information stored in the information processing system50from the information processing system50, and displays object based on the object information on the display480.

The code generation unit35encodes the device ID of the electronic whiteboard2stored in a device information storage area3001and information indicating that the device is usable in the conference into a two-dimensional pattern, to generate a two-dimensional code. The code generation unit35may encode, into a barcode, the device ID of the electronic whiteboard2and the information indicating that the electronic whiteboard2is a device usable in the conference. The device ID is, for example, either a serial number or a universally unique identifier of the electronic whiteboard2. The device identification information may be set by the user. Note that the code generation unit35also serves as an output unit that outputs a two-dimensional code or a barcode.

The electronic whiteboard2also includes a storage unit3000implemented by the SSD404or the like illustrated inFIG.8. The storage unit3000includes the device information storage area3001and the object information storage area3002each of which is implemented by a database, for example.

FIG.14illustrates information such as device ID stored in the device information storage area3001.

The item “device ID” is identification information identifying the electronic whiteboard2.

The item “Internet Protocol (IP) address” is used by another device to connect to the electronic whiteboard2via a network.

The item “password” is used for authentication performed when another apparatus connects to the electronic whiteboard2.

FIG.15illustrates an example of object information stored in the object information storage area3002according to the present embodiment. The object information is information for managing an object displayed by the electronic whiteboard2. The object information is transmitted to the information processing system50and is used as minutes.

In a case where the electronic whiteboard2is located at the second site when the teleconference is held, the object information is shared with the first site.

The item “conference ID” indicates identification information of a conference notified from the information processing system50.

The item “object ID” indicates identification information for identifying an object.

The item “type” indicates a type of the object. the type of object includes, for example, handwriting, text, graphic, and image. “Handwriting” represents stroke data (coordinate point sequence). “Text” represents a character string (character codes) input from a software keyboard. The character string may also be referred to as text data. “Graphic” is a geometric shape such as a triangle or a quadrangle. “Image” represents image data in a format such as Joint Photographic Experts Group (JPEG), Portable Network Graphics (PNG), or Tagged Image File Format (TIFF) acquired from, for example, a PC or the Internet.

A single screen of the electronic whiteboard2is referred to as a page. A “page” indicates the page number.

The item “coordinates” indicate a position of an object relative to a predetermined origin on the electronic whiteboard2. The position of the object is, for example, the upper left vertex of a circumscribed rectangle of the object. The coordinates are expressed, for example, in units of pixels of the display.

The item “size” indicates a width and a height of the circumscribed rectangle of the object.

Screen Transition

Descriptions are now given of several screens displayed by the communication terminal10in a teleconference, with reference toFIGS.16to19.FIG.16is a diagram illustrating an example of an initial screen200displayed by the information recording application41operating on the communication terminal10after a login. The user of the communication terminal10connects to the information processing system50on the information recording application41. The user inputs authentication information, and when the login is successful, the initial screen200ofFIG.16is displayed.

The initial screen200includes a fixed display button201, a change front button202, the panoramic image203, one or more talker images204ato204c, and a start recording button205. In the following description, each of the talker images204ato204cmay be simply referred to as a “talker image204,” when not distinguished from each other. In a case where the meeting device60has already been started and is capturing an image of the surroundings at the time of the login, the panoramic image203and the talker images204created by the meeting device60are displayed on the initial screen200. This allows the user to decide whether to start recording while viewing the panoramic image203and the talker images204. In a case where the meeting device60is not started (is not capturing any image), the panoramic image203and the talker images204are not displayed.

The information recording application41may display the talker images204of all participants based on all faces detected from the panoramic image203, or may display the talker images204of certain number (N) of persons who have made an utterance most recently. In the example illustrated inFIG.16,the talker images204of up to three persons are displayed. Display of the talker image204of a participant may be omitted until one of the participants makes an utterance (in this case, the number of the talker images204increases by one in response to an utterance). Alternatively, the talker images204of three participants in a predetermined direction may be displayed (the talker images204are switched in response to an utterance).

When no participant is speaking such as immediately after the meeting device60is turned on, an image of a predetermined direction (such as 0 degrees, 120 degrees, or 240 degrees) of 360 degrees in the horizontal direction is generated as the talker image204. When fixed display (described later) is set, the setting of the fixed display is prioritized.

The fixed display button201is a button for the user to perform an operation of fixing a certain area of the panoramic image203as the talker image204in close-up.

The change front button202is a button for the user to perform an operation of changing the front of the panoramic image203. Since the panoramic image presents the 360-degree surroundings in the horizontal direction, the right end and the left end matches to the same direction. The user slides the panoramic image203leftward or rightward with a pointing device to set a particular participant to the front. The user’s operation is transmitted to the meeting device60. The meeting device60changes the angle set as the front in 360 degrees in the horizontal direction, creates the panoramic image203, and transmits the panoramic image203to the communication terminal10.

When the user presses the start recording button205, the information recording application41displays a recording setting screen210illustrated inFIG.17.

FIG.17is a diagram illustrating an example of the recording setting screen210displayed by the information recording application41. The recording setting screen210allows the user to set whether to record (whether to include in a recorded video) the panoramic image203and the talker images204created by the meeting device60and the desktop screen of the communication terminal10or the screen of the application operating on the communication terminal10. In a case where the information recording application41is set to record none of the panoramic image, the talker image, and the desktop screen or the screen of the operating application, the information recording application41records only audio (audio output by the communication terminal10and audio collected by the meeting device60).

A camera toggle button211is a button for switching on and off of recording of the panoramic image and the talker image generated by the meeting device60. Alternatively, the camera toggle button211may allow settings for switching on and off of recording of the panoramic image and the talker image individually.

A PC screen toggle button212is a button for switching on and off of recording of the desktop screen of the communication terminal10or a screen of an application operating on the communication terminal10. When the PC screen toggle button212is on, the desktop screen is recorded.

When the user desires to record the screen of the application, the user further selects the application in an application selection field213. In the application selection field213, names of applications operating on the communication terminal10are displayed in a pull-down format. Thus, the application selection field213allows the user to select an application whose screen is to be recorded. The information recording application41acquires the names of the applications from the OS. The information recording application41can display names of applications that have a user interface (UI) (screen) among applications being executed. The applications to be selected may include the teleconference application42. Thus, the information recording application41can record a material displayed by the teleconference application42, the participant at each site, and the like as a video. In addition, various applications such as a presentation application, a word processor application, a spreadsheet application, and a Web browser application are displayed in a pull-down manner. This thus allows the user to flexibly select the screen of the application to be included in the combined video.

When recording is performed in units of applications, the user is allowed to select a plurality of applications. The information recording application41can record the screens of all the selected applications.

When both the camera toggle button211and the PC screen toggle button212are set to off, a message “Only audio is recorded” is displayed in a recorded content confirmation window214. The audio in this case includes audio output from the communication terminal10(audio received by the teleconference application42from the second site101) and audio collected by the meeting device60. That is, when a teleconference is being held, the audio from the teleconference application42and the audio from the meeting device60are stored regardless of whether the images are recorded. Note that the user may make a setting to selectively stop storing the sound from the teleconference application42and the sound from the meeting device60according to user settings.

In accordance with a combination of on and off of the camera toggle button211and the PC screen toggle button212, a combined video is recorded in the following manner. The combined video is displayed in real time in the recorded content confirmation window214.

In a case where the camera toggle button211is on and the PC screen toggle button212is off, the panoramic image and the talker images created by the meeting device60are displayed in the recorded content confirmation window214.

If the camera toggle button211is off and the PC screen toggle button212is on (and the screen has also been selected), the desktop screen or the screen of the selected application is displayed in the recorded content confirmation window214.

In a case where the camera toggle button211is on and the PC screen toggle button212is on, the panoramic image and the talker images created by the meeting device60and the desktop screen or the screen of the selected application are displayed side by side in the recorded content confirmation window214.

Thus, an image created by the information recording application41is referred to as a combined video for convenience in the present embodiment although there is a case where the panoramic image and the talker images or the screen of the application is not recorded or a case where none of the panoramic image, the talker image, and the screen of the application are recorded.

The recording setting screen210further includes a check box215labelled as “automatically transcribe after uploading the record.” The recording setting screen210further includes a button216labelled as “start recording now.” If the user checks the check box215, text data converted from utterances made during the teleconference is attached to the recorded video. In this case, after the end of recording, the information recording application41uploads audio data to the information processing system50together with a text data conversion request. When the user presses the button216labelled as “start recording now,” a recording-in-progress screen220is displayed as illustrated inFIG.18.

FIG.18is an example of the recording-in-progress screen220displayed by the information recording application41during recording. In the description referring toFIG.18,for simplicity, mainly differences fromFIG.16are described. The recording-in-progress screen220displays, in real time, the combined video being recorded according to the conditions set by the user in the recording setting screen210. The recording-in-progress screen220inFIG.18corresponds to the case where the camera toggle button211is on and the PC screen toggle button212is off, and displays the panoramic image203and the talker images204(both are moving images) created by the meeting device60. The recording-in-progress screen220includes a recording icon225, a pause button226. and a stop recording button227.

The pause button226is a button for pausing the recording. The pause button226also receives an operation of resuming the recording after the recording is paused. The stop recording button227is a display component (visual representation) for receiving an instruction for ending the recording. The recording ID is does not changed when the pause button226is pressed, whereas the recording ID is changed when the stop recording button227is pressed. After pausing or temporarily stopping the recording, the user allowed to set the recording conditions set in the recording setting screen210again before resuming the recording or starting recording again. In this case, the information recording application41may generate multiple video files each time the recording is stopped (e.g., when the stop recording button227is pressed), or may consecutively combine the plurality of video files to generate a single video (e.g., when the pause button226is pressed). When the information recording application41plays the combined video, the information recording application41may play the plurality of recorded files continuously as one video.

The recording-in-progress screen220includes a button221labelled as “get information from calendar,” a conference name field222, a time field223, and a location field224. The button221labelled as “get information from calendar” allows the user to acquire conference information from the conference management system9. When the user presses the button221labelled as “get information from calendar,” the information recording application41acquires a list of conferences for which the user has a viewing authority from the information processing system50and displays the acquired list of conferences. The user selects a teleconference to be held from the list of conferences. Consequently, the conference information is reflected in the conference name field222, the time field223, and the location field224. The title, the start time and the end time, and the location included in the conference information are reflected in the conference name field222, the time field223, and the location field224, respectively. The conference information and the record in the conference management system9are associated with each other by the conference ID.

In response the user ending the recording after the end of the teleconference, a combined video with sound is created.

FIG.19is an example of a conference list screen230displayed by the information recording application41. The conference list screen230presents a list of conferences, specifically, a list of the records (videos) recorded during teleconferences. The list of conferences includes conferences held in a certain conference room as well as teleconferences.

The conference list screen230displays conference information for which the logged-in user has a right to view, in the conference information storage area5001. The information on the video, stored in the information storage area1001, may be further integrated.

The conference list screen230is displayed when the user selects a conference list tab231on the initial screen200ofFIG.16. The conference list screen230displays a list236of the videos (records) for which the user has the viewing authority. The conference creator (minutes creator) can set the right to view for a participant of the conference. The list of conferences may be a list of stored records, a list of scheduled conferences, or a list of conference data.

The conference list screen230includes items of a check box232, an update date/time233, a title234. and a status235.

The check box232receives selection of a video file. The check box232is used when the user desires to collectively delete video files.

The update date/time233indicates a recording start time of the combined video. If the combined video is edited, the update date/time233may indicate the edited date and time.

The title234indicates the title (such as a subject) of the conference. The title may be transcribed from the conference information or set by the user.

The status235indicates whether the combined video has been uploaded to the information processing system50. If the video has not been uploaded, “local PC” is displayed, whereas if the video has been uploaded, “uploaded” is displayed. If the video has not been uploaded, an upload button is displayed. If there is a combined video yet to be uploaded, it is desirable that the information recording application41automatically upload the combined video when the user logs into the information processing system50.

When the user selects a desired title from the list236of the combined videos with a pointing device, the information recording application41displays a replay screen. The replay screen allows playback of the combined video.

It is desirable that the information recording application41provides a function for the user to narrow down conferences based on the update date and time, the title, the keyword, or the like. Further, there may be a where the user has a difficulty finding a conference of interest because many conferences are displayed. For such a case, the information recording application41desirably provides a search function for receiving input of a word or phrase to narrow down the video (record) and to present videos having a title or including an utterance that matches the input word or phrase. The search function allows the user to find desired record in a short time even if the number of records increases. The conference list screen230may allow the user to sort the conferences by using the update date and time or the title.

Operation or Processing

FIG.20is a sequence diagram illustrating an operation performed by the record creation system100to associate the meeting device60with the electronic whiteboard2. In the description ofFIG.20, a user who participates in a conference using the meeting device60and the electronic whiteboard2in the same conference.

S1: When the electronic whiteboard2installed in the conference room in which the conference is to be held is powered on, the electronic whiteboard2communicates with the preset information processing system50. The electronic whiteboard2specifies the device ID and registers that the electronic whiteboard2can be associated with the conference.

S2: The code generation unit35of the electronic whiteboard2disposed in the conference room and to be used in the conference generates a two-dimensional code in which the device ID of the electronic whiteboard2and information indicating that the device is usable in the conference. The display control unit34displays the two-dimensional code. The two-dimensional code may further include a password for the electronic whiteboard2to authenticate the other device.

S3: The user carrying the communication terminal10and the meeting device60enters the conference room where the electronic whiteboard2is installed, and connects the communication terminal10and the meeting device60with a USB cable. The meeting device60starts up in response to power supply from the USB cable or power-on. In this way, the meeting device60enters a standby state. The user starts the information recording application41on the communication terminal10. The information recording application41starts communicating with the meeting device60, so that the meeting device60starts capturing images and collecting sound. The panoramic image generation unit62of the meeting device60captures an image of the surroundings and generates a panoramic image of the surroundings (image data) including the two-dimensional code.

S4: The terminal communication unit61of the meeting device60transmits the panoramic image and talker images to the communication terminal10.

S5: The device communication unit16of the communication terminal10receives the panoramic image. The code analysis unit22detects the two-dimensional code displayed on the electronic whiteboard2from the panoramic image. The code analysis unit22decodes the two-dimensional code. If the code analysis unit22determines that information indicating that the device is usable in the conference is embedded, the code analysis unit22acquires the device ID of the electronic whiteboard2from the two-dimensional code. The two-dimensional code may be analyzed by the meeting device60. That is, the meeting device60may include a code analysis unit.

S6: The communication unit11implemented by the information recording application41specifies the device ID of the electronic whiteboard2and transmits a registration request for a conference to the information processing system50. Preferably, the communication unit11further transmits identification information of the meeting device60to the information processing system50.

S7: When the communication unit51of the information processing system50receives a registration request (device ID) for a conference, the communication management unit54issues a conference ID. In a case where the information recording application41receives the selection of the conference from the conference list screen230or the like, the communication unit51attaches the conference ID to the device ID. Accordingly, the communication management unit54does not issue the conference ID.

S8: Then, the device management unit55stores the device ID of the electronic whiteboard2and the conference ID in association with each other (and preferably the device ID of the meeting device60and the conference ID in association with each other) in the association storage area5003.

S9, S10: The communication unit51of the information processing system50transmits the conference ID to the communication terminal10and the electronic whiteboard2. The communication unit11of the communication terminal10receives and stores the conference ID. Similarly, when the communication unit36of the electronic whiteboard2receives the conference ID, the communication unit36stores the conference ID. The communication terminal10receives at least one of the conference ID and the device ID as a response to the registration request for the conference. The electronic whiteboard2and the information processing system50may communicate with each other by a two-way communication scheme such as WebSocket that enables push communication from the information processing system50to the electronic whiteboard2.

Since the electronic whiteboard2and the communication terminal10have the same conference ID, the electronic whiteboard2and the meeting device60are associated with the conference. After that, the communication terminal10attaches at least one of the conference ID and the identification information of the meeting device60to data to be transmitted, and the electronic whiteboard2attaches at least one of the conference ID and the device ID to data to be transmitted. In this manner, the conference ID is attached to the communication in the present embodiment. Alternatively, the device ID or the identification information of the meeting device60may be attached to the communication. The information processing system50can specify the conference ID from the attached identification information based on the association information.

The associating the electronic whiteboard2with the conference ID inFIG.20is performed for associating the record with the hand-drafted data or the like input to the electronic whiteboard2. Further, the electronic whiteboard2has a function of sharing objects with another electronic whiteboard2located at a remote site. The electronic whiteboard2and another electronic whiteboard2are registered in a server for teleconference and can share hand-drafted data or the like in real time.

FIG.21is a flowchart illustrating the operation of the information processing system50. The communication unit51of the information processing system50monitors whether a device ID is received from the communication terminal10until a device ID is received (S101).

When the communication unit51receives the device ID (Yes in S101), the communication management unit54issues a conference ID (S102). When the communication unit51receives the conference ID attached to the device ID, the communication management unit54does not issue the conference ID.

The device management unit55stores the conference ID and the received device ID in association with each other in the association storage area5003(S103). The device management unit55maintains the association between the conference ID and the device ID until the end of the conference (end of recording).

The communication unit51of the information processing system50transmits the conference ID to the communication terminal10and the electronic whiteboard2(S104).

FIG.22is a flowchart illustrating the operation of the information recording application41operating on the communication terminal10. This process is performed for a case where there is an obstacle (e.g., a person) in front of the electronic whiteboard2to inhibit the information recording application41from detecting the two-dimensional code, and accordingly the electronic whiteboard2cannot be associated with the conference.

First, the information recording application41is activated (S111).

Next, the time measuring unit25starts measuring the time from the activation of the information recording application41to the detection of the two-dimensional code (S112).

Since the device communication unit16receives the panoramic image, the code analysis unit22detects the two-dimensional code. The time measuring unit25determines whether or not the two-dimensional code is detected within a predetermined time after the activation (S113).

When the two-dimensional code is detected within the predetermined time after the activation (Yes in S113), the time measuring unit25stops measuring time (S114).

When the two-dimensional code is not detected within the predetermined time after the activation (No in S113), the display control unit13displays an error dialog in response to a notification from the time measuring unit25(S115).

FIG.23is an example of a message301displayed in the error dialog by the information recording application41executing on the communication terminal10. The message301of the error dialog is, for example, “Two-dimensional code is not detected. Please confirm that the two-dimensional code is displayed and that there is no person in front of the electronic whiteboard.” The user can recognize from the message301that the two-dimensional code is not detected, and can take measures such as causing a person to move away from the front of the electronic whiteboard2. An OK button302is a button for the information recording application41to measure the time again.

Referring back toFIG.22, when the pressing of the OK button302is received in step S115and the error dialog is closed (Yes in step S116), the process returns to step S112. The time measuring unit25starts measuring time.

For the user to close the error dialog, the error dialog may be provided with a cancel button so that the user can start the conference without associating the electronic whiteboard2with the conference.

Two-Dimensional Code and Barcode Displayed by Electronic Whiteboard

A supplemental description is given of the two-dimensional code and the barcode displayed by the electronic whiteboard2, with reference toFIGS.24to31. If there is an obstacle69between the meeting device60and the electronic whiteboard2, the communication terminal10may not be able to acquire the device ID. Therefore, it is effective to adjust the position at which the electronic whiteboard2displays the two-dimensional code8as described below.

FIG.24is a schematic diagram illustrating an example of the position of the two-dimensional code8displayed on the electronic whiteboard2. As described above, the two-dimensional code8includes the device ID and the information indicating a device usable in a conference. On the desk where the meeting device60is placed, there is the obstacle69between the meeting device60and the electronic whiteboard2. Note that the camera included in the meeting device60is not necessarily a spherical or hemispherical camera for detecting the two-dimensional code8. The electronic whiteboard2displays the device ID above a center line320of the display screen of the electronic whiteboard2in the vertical direction. Thus, even if the obstacle69is present between the meeting device60and the electronic whiteboard2, the meeting device60can easily capture an image of the device ID.

As illustrated inFIG.25, the electronic whiteboard2may display the center of the two-dimensional code8above the center line320of the screen of the electronic whiteboard2in the vertical direction.

Alternatively, as illustrated inFIG.26, the electronic whiteboard2may move the two-dimensional code8according to an elapsed time. InFIG.26, the two-dimensional code8moves from left to right. The electronic whiteboard2may move the two-dimensional code8being displayed. Alternatively, the electronic whiteboard2may display the two-dimensional code8, stop displaying the two-dimensional code8, and again display the two-dimensional code8at a different position. Thus, even if the position of the obstacle69is uncertain, the meeting device60can easily capture the device ID. Further, even when the electronic whiteboard2displays the two-dimensional code8in a small size in order to reduce the feeling of pressure on the user, the meeting device60can easily capture the image of the two-dimensional code8.

Furthermore, the electronic whiteboard2may change the size of the two-dimensional code8while moving the two-dimensional code8.

Alternatively, as illustrated inFIG.27, the electronic whiteboard2may simultaneously display a plurality of two-dimensional codes8. Thus, even if some of the two-dimensional codes8are hidden by the obstacle69. the meeting device60can easily capture images of the other two-dimensional codes8. All of the two-dimensional codes8may include the same set of information or different sets of information.

Alternatively, as illustrated inFIG.28, the electronic whiteboard2may display the two-dimensional code8at a position adjacent to (close to) a menu71. The menu71extends vertically at the right end. Similar toFIG.24, the two-dimensional code8is displayed above the center line320of the screen of the electronic whiteboard2in the vertical direction.

The two-dimensional code8displayed close to the menu71is less likely to cause discomfort for the user. In addition, This allows the user to use a wide area of the screen.

Alternatively, as illustrated inFIG.29, the electronic whiteboard2may display the two-dimensional code8in the menu71. Compared with the example ofFIG.28, this configuration alleviates the discomfort for the user and allows the user to use a wider area of the screen.

FIG.30is a schematic diagram illustrating an example of a method for displaying the two-dimensional code8performed by the electronic whiteboard2in a case where the meeting device60includes a hemispherical camera. By using a hemispherical camera having a wide field of view in the horizontal direction in the configurations illustrated inFIGS.24to29, the two-dimensional code8can be found more easily.

Note that a barcode can be displayed in the same manner as the manner of display of the two-dimensional code8inFIGS.24to30.

FIG.31is a schematic diagram illustrating an example of a method for displaying a barcode7by the electronic whiteboard2.

Note that the barcode7is less robust against inclination than the two-dimensional code8. For this reason, the code analysis unit22implemented by the information recording application41cuts out a monochrome pattern of the barcode7and adjusts a skew angle and a pitch angle of the monochrome pattern. The code analysis unit22performs edge enhancement on black bars. The code analysis unit22performs pattern matching of the cut out image with a pattern (from a start character to a stop character at the right end) registered as a pattern of the barcode7, so as to detect the barcode7on the electronic whiteboard2.

The examples in which the two-dimensional code8or the barcode7is displayed have been described with reference toFIGS.24to31. Alternatively, the information recording application41may detect, by using optical character recognition (OCR), a device ID (alphabets or numbers) displayed by the electronic whiteboard2.

Storage of Combined Video

A process of storing a combined video will be described with reference toFIG.32.FIG.32is a sequence chart illustrating an example of recording a panoramic image, a talker image, and an application screen by the information recording application41. Since the conference ID is transmitted from the information processing system50to the communication terminal10and the electronic whiteboard2in S9and S10ofFIG.20, the information recording application41can record the video.

S21: The user operates the teleconference application42to start a teleconference. In this example, the teleconference application42at the first site102and the teleconference application42at the second site101start a teleconference. The teleconference application42operating on the communication terminal10at the first site102transmits an image captured by the camera of the meeting device60and audio collected by the microphone of the meeting device60to the teleconference application42operating on the communication terminal10at the second site101The teleconference application42on the communication terminal10at the second site101displays the received image on the display of the communication terminal10and outputs the received audio from the speaker of the communication terminal10. Similarly, the teleconference application42on the communication terminal10at the second site101transmits an image captured by a camera of another meeting device60at the second site101and audio collected by a microphone of the meeting device60at the second site101to the teleconference application42on the communication terminal10at the first site102. The teleconference application42on the communication terminal10at the first site102displays the received image on the display and outputs the received audio from the speaker. The teleconference application42at the first site102and the teleconference application42at the second site101repeat this processing to implement the teleconference.

S22: The user inputs settings relating to recording on the recording setting screen210illustrated inFIG.17, provided by the information recording application41. The operation reception unit12implemented by the information recording application41receives the settings. In this example, both the camera toggle button211and the PC screen toggle button212are on.

In a case that the user has reserved a teleconference in advance, a list of teleconferences is displayed in response to pressing of the button221labeled as “get information from calendar” illustrated inFIG.18by the user. The user selects a desired teleconference to be associated with the video to be recorded. Since the user has logged into the information processing system50, the information processing system50identifies teleconferences for which the logged-in user has a right to view. The information processing system50transmits the list of the identified teleconferences to the communication terminal10. Thus, the user selects a teleconference that is being held or to be held. In this way, information related to the teleconference such as the conference ID is determined. In the present embodiment, the conference ID has already been generated in the processing illustrated inFIG.20.

S23: The user instructs the information recording application41to start recording. For example, the user presses the button216labelled as “start recording now.” The operation reception unit12implemented by the information recording application41receives the instruction. The display control unit13displays the recording-in-progress screen220.

S24: Since the conference ID is determined, the communication unit11implemented by the information recording application41specifies the conference ID and requests the information processing system50to transmit information on the storage location.

S25: The communication unit51of the information processing system50receives the request. The communication management unit54transmits information on the storage location (URL of the storage service system70) of the combined video (video file) to the information recording application41via the communication unit51.

S26: When the communication unit11implemented by the information recording application41receives the conference ID and the storage location of the video file, the recording control unit17determines that preparation for recording is completed and starts recording.

S27: The app screen acquisition unit14implemented by the information recording application41request an application selected by the user to send a screen thereof. More specifically, the app screen acquisition unit14acquires the screen of the application via the OS. The description given with reference toFIG.32is on the assumption that the user selects the teleconference application42.

S28: The recording control unit17implemented by the information recording application41notifies the meeting device60of the start of recording via the device communication unit16. With the notification, the recording control unit17preferably sends information indicating that the camera toggle button211is on (a request for a panoramic image and a talker image). The meeting device60transmits the panoramic image and the talker image to the information recording application41regardless of the presence or absence of the request.

S29: In response to receiving the notification of the start of recording by the terminal communication unit61of the meeting device60, a unique recording ID is assigned. The terminal communication unit61transmits the recording ID to the information recording application41. In one example, the information recording application41assigns the recording ID. In another example, the recording ID is acquired from the information processing system50.

S30: The audio reception unit15implemented by the information recording application41acquires audio data output by the communication terminal10(audio data received by the teleconference application42).

S31: The device communication unit16transmits the audio data acquired by the audio reception unit15and a combining request of audio to the meeting device60.

S32: In response to receiving the audio data and the combining request by the terminal communication unit61of the meeting device60, the audio synthesis unit65combines (or synthesizes) the received audio data with the audio of the surroundings collected by the sound collection unit64. For example, the audio synthesis unit65adds the two audio data items together. Since clear sound around the meeting device60is recorded, particularly the accuracy of text converted from the sound around the meeting device60(in the conference room) increases.

The communication terminal10may perform this combination of the audio data. Alternatively, the recording function may be allocated to the meeting device60, and the audio processing may be allocated to the communication terminal10. In this case, the load on the meeting device60is reduced.

S33: Further, the panoramic image generation unit62of the meeting device60generates a panoramic image, and the talker image generation unit63generates a talker image.

S34: The device communication unit16of the information recording application41repeatedly receives the panoramic image (surrounding image data) and the talker image from the meeting device60. Further, the device communication unit16repeatedly receives the combined audio data from the meeting device60. The device communication unit16may send a request to the meeting device60to acquire such images and data. Alternatively, the meeting device60that has received information that the camera toggle button211is on may automatically transmit the panoramic image and the talker image. In response to receiving the combining request of audio, the meeting device60may automatically transmit the combined audio data to the information recording application41.

S35: The recording control unit17implemented by the information recording application41arranges the application screen acquired from the teleconference application42, the panoramic image203. and the talker images204adjacent with one another, to create a combined image. The recording control unit17repeatedly creates the combined image and designates each combined image as a frame of a video, to create a combined video. The recording control unit17stores the audio data received from the meeting device60.

S36: The communication unit36of the electronic whiteboard2transmits the object information (information on, for example, hand-drafted data) to the information processing system50in association with the conference ID, preferably for each stroke.

The processing of steps S30to S36is not necessarily performed in the order presented inFIG.32. For example, the combining the audio data and the creating the combined image may be performed in opposite order.

As described above, in the record creation system100according to the present embodiment, the meeting device60acquires the device ID of the device used in the conference room, and the communication terminal10transmits the device ID to the information processing system50. Then, the information processing system50associates the conference with the device used in the conference. Since the meeting device60is connected to the communication terminal10that communicates with the information processing system50, the meeting device60is also associated with the conference. This configuration obviates the trouble of the user of capturing the two-dimensional code displayed by the device with the camera or registering the device and the meeting device60in the information processing system50for a conference. With this configuration, a plurality of devices (the meeting device60and the electronic whiteboard2) can be associated with the conference with minimum user intervention.

In the present embodiment, the electronic whiteboard2that outputs the device ID by sound and the processing thereof will be described.

Note that the present embodiment is described on an assumption that the hardware configurations ofFIGS.4,5, and8described in the above embodiment are applicable.

Functions

FIG.33is a block diagram illustrating functional configurations of the communication terminal10, the meeting device60, and the information processing system50of the record creation system100according to the present embodiment. In the following description with reference toFIG.33, the differences fromFIG.9are mainly described.

InFIG.33, the communication terminal10includes a sound analysis unit23instead of the code analysis unit22. As will be described later, the sound analysis unit23performs spectrum analysis on sound data to detect a frequency included in the sound data, and converts a specific frequency into bit data, to acquire the device ID.

FIG.34is a block diagram illustrating the functional configuration of the electronic whiteboard2according to the present embodiment. In the following description with reference toFIG.34, differences fromFIG.13are mainly described. The electronic whiteboard2illustrated inFIG.34includes a sound data generation unit37. The sound data generation unit37converts each of alphabets and numerals in the device ID into a frequency of time T, and generates a sound signal by sampling at a constant interval similar to pulse code modulation (PCM) conversion. The sound signal is converted into an analog signal by an analog-to-digital (A/D) converter included in the speaker450(seeFIG.8) and output from the speaker450. The sound data generation unit37serves as an output unit that outputs a sound signal.

In the present embodiment, the sound collection unit64of the meeting device60serves as an acquisition unit that acquires the sound data.

Generation of Sound Data

FIGS.35A to35Care diagrams illustrating examples of a frequency of sound and a bit pattern used for transmission of the device ID. In FIG.FIGS.35A to35C,the frequencies of sound used for transmitting the device ID are 18 kHz, 19 kHz, and 20 kHz. The frequency of 18 kHz is used for a pilot signal to be added before the message. The length of the time T corresponds to one bit. The frequency of 19 kHz corresponds to 0, and the frequency of 20 kHz corresponds to 1. Therefore,FIG.35Aillustrates a value of 00,FIG.35Billustrates a value of 01,FIG.35Cillustrates a value of 10, andFIG.35Dillustrates a value of 11.

The pilot signal is transmitted by adding 2 bits to every 8 bits of data. When the sound signal represents one alphabet or one numeral by 8 bits, 10 bits are used to transmit one character of the device ID. When the electronic whiteboard2transmits, for example, an 8-bit American Standard Code for Information Interchange (ASCII) character “e” (01100101 in binary, 0×65 in hexadecimal), the frequency pattern of the sound signal is as illustrated inFIG.36.

Numerals 0 to 9 and English capital letters A to Z are used for the device ID of the electronic whiteboard2. Therefore, the device ID is represented by 8-bit ASCII codes. One character of the device ID has the frequency pattern as illustrated inFIG.36together with the pilot signal. In the electronic whiteboard2, frequency patterns corresponding to the numerals 0 to 9 and the characters A to Z and sound signals based on the time T for one bit are encoded by PCM and stored in advance. For each character of the device ID of the electronic whiteboard2, the sound data generation unit37converts the PCM-encoded sound data into digital sound data using a sound source.

Since the sound collection unit64of the meeting device60collects the ambient sound and converts the collected ambient sound into sound data (sound signals), the sound collection unit64also collects the sound signal generated by the sound data generation unit37. The sound analysis unit23of the communication terminal10performs spectrum analysis (Fourier transformation) on the sound signal at regular time intervals (for example, several tens of milliseconds), to obtain a spectrum having a peak at 18 kHz, 19 kHz, or 20 kHz. The sound analysis unit23detects the head of a character string used as the device ID with the frequency of 18 kHz and converts the frequency of 19 kHz or the 20 kHz into the value of 0 or 1. As a supplementary explanation, the time (for example, several tens of milliseconds) for the spectrum analysis is shorter than the time T. Accordingly, the sound analysis unit23combines, for each time T, the sound data into one bit of 0 or 1 depending on which of 0 and 1 is greater, and reproduces the device ID. The time (for example, several tens of milliseconds) for the spectrum analysis may be the same as the time T.

A description is given below of a sequence of operations.

FIG.37is a sequence diagram illustrating an operation performed by the record creation system100. In the following description with reference toFIG.37, differences fromFIG.20are mainly described. Step S41is similar to Step S1inFIG.20.

S42: The user presses a button on the electronic whiteboard2placed in the conference room used in the conference for outputting the device ID of the electronic whiteboard2. The sound data generation unit37of the electronic whiteboard2generates a sound signal representing, by frequency, the device ID of the electronic whiteboard2and outputs the sound signal from the speaker450(seeFIG.8). The sound signal may be automatically output at power-on.

S43: The sound collection unit64of the meeting device60collects the sound signal output by the electronic whiteboard2with a microphone and performs PCM encoding on the sound signal.

S44: The terminal communication unit61of the meeting device60transmits the sound signal to the communication terminal10.

S45: The device communication unit16of the communication terminal10receives the sound signal. The sound analysis unit23performs frequency analysis on the sound signal, to divide the sound signal for each pilot signal of 18 kHz, and converts the frequencies (19 kHz, 20 kHz) included in each divided piece of the sound signal into an 8-bit string based on the conversion rule ofFIGS.35A to35D. The sound analysis unit23converts the sound data into the device ID by using a conversion table of 8-bit ASCII codes. In the conversion table, numerals 0 to 9 and English capital letters A to Z (8-bit strings) are associated with the frequency patterns illustrated inFIGS.36A to36D.

S46: The communication unit11of the communication terminal10transmits the device ID to the information processing system50. Subsequent processing may be similar to the processing inFIG.20.

Although the meeting device60collects the sound signal inFIG.37, the communication terminal10may directly collect the sound signal.

In addition, although the frequencies of the sound are set to 18 kHz to 20 kHz in the above-described embodiment, these frequencies are in an audible range, and there is a concern that the user may hear the sound. Thus, for example, the electronic whiteboard2may output the device ID as a sound with ultrasonic waves of about 50 to 100 kHz.

FIGS.38A to38Dillustrates examples of correspondence between frequencies and bit data in the case of ultrasonic waves. InFIGS.38A to38D,ultrasonic waves having frequencies of 50 kHz (used as a pilot signal), 55 kHz, and 60 kHz are illustrated, but the ultrasonic waves may have other frequencies.FIG.38Aillustrates a value of00,FIG.38Billustrates a value of01,FIG.38Cillustrates a value of 10, andFIG.38Dillustrates a value of 11.

In this case, it is preferable that the speaker450of the electronic whiteboard2also supports ultrasonic waves and that the microphones608of the meeting device60supports ultrasonic waves.

According to the present embodiment, the electronic whiteboard2can notify the meeting device60of the device ID by sound. Accordingly, the present embodiment provides, in addition to the effects of Embodiment 1, an effect of making it easier for the meeting device60to acquire the device ID even if a person is present in front of the electronic whiteboard2.

In the present embodiment, processing at the end of recording (end of conference) will be described. When the user ends the recording, the information processing system50deletes the association between the conference ID and the device ID (releases the electronic whiteboard2). Further, the electronic whiteboard2displays the end of the recording and resumes displaying or outputting the device ID.

The hardware configuration illustrated inFIGS.4,5, and8and the functional configuration illustrated inFIG.9of the above-described embodiment are applicable to the present embodiment.

Operation or Processing

FIG.39is a sequence diagram illustrating processing performed by the record creation system100when ending the recording of a conference.

S51: ThXe user is about to end the conference in which the meeting device60and the electronic whiteboard2are used. The user presses the stop recording button227on the information recording application41. The operation reception unit12receives the pressing operation.

S52: The recording control unit17implemented by the information recording application41stops recording the video (stops creating the combined video) and stops recording the audio.

S53: The communication unit11implemented by the information recording application41transmits a notification of the end of the conference (conference end notification) to the information processing system50with designation of the conference ID.

S54: The communication unit51of the information processing system50receives the conference end notification. The communication management unit54transmits the conference end notification to the electronic whiteboard2that communicates with the information processing system50with the designation of the conference ID of the conference to be ended.

S55: The communication unit36of the electronic whiteboard2receives the conference end notification, and the display control unit34displays a conference end notification screen.

FIG.40illustrates an example of a conference end notification screen310displayed by the electronic whiteboard2. The conference end notification screen310includes a message311, an end button312(an example of a display component), and a continue button313. The message311is for example, “The conference (recording of information) is to end. After that, handwriting will not be included in the record. Please press “END” to end the conference or “CONTINUE” to continue the conference.” In response to receiving the user’s pressing the end button312(an instruction of ending the recording), the electronic whiteboard2ends the recording of the object displayed thereon (the object information is not transmitted to the information processing system50). In response to receiving the user’s pressing the continue button313, the electronic whiteboard2records (transmit, to the information processing system50,) the object added by the user to the electronic whiteboard2.

S56: When the user presses the end button312on the conference end notification screen310displayed by the electronic whiteboard2, the contact position detection unit31receives the pressing.

S57: The communication unit36of the electronic whiteboard2designates the conference ID and transmits an acknowledgment of conference end to the information processing system50. The communication unit36ends the transmission of the object information to the information processing system50.

S58: When the communication unit51of the information processing system50receives the acknowledgment of conference end, the device management unit55deletes the association (association information) between the conference ID and the device ID.

S59: The communication unit51of the information processing system50transmits a notification of association cancel completion to the electronic whiteboard2.

S60: The communication unit36of the electronic whiteboard2receives the notification of association cancel completion. Then, the electronic whiteboard2resumes the output of the two-dimensional code or the barcode in the case of Embodiment1, or the sound signal in the case of Embodiment2. The data recording unit33deletes the conference ID.

S61: The communication unit51of the information processing system50transmits the notification of association cancel completion to the communication terminal10. The communication unit11implemented by the information recording application41receives the notification of association cancel completion and deletes the conference ID.

S62: In response to receiving the notification of association cancel completion, the device communication unit16implemented by the information recording application41transmits a recording end notification to the meeting device60. The meeting device60continues creating the panoramic image and the talker image and combining the audio. The meeting device60may change the processing, for example, changing the resolution or frame rate depending on whether or not recording is being performed. The meeting device60may interrupt the creation of the panoramic image and the talker image or the combining of the audio in a case where the information recording application41is not operated for a predetermined period, for example.

S63: The recording control unit17implemented by the information recording application41combines the audio data with the combined video, to create the combined video with sound.

S64: In a case that the user puts a mark in the check box215labelled as “automatically transcribe after uploading the record” on the recording setting screen210. the audio data processing unit18requests the information processing system50to convert the audio data into text data.

Specifically, the audio data processing unit18designates the URL of the storage location, and transmits, via the communication unit11, a request to convert the audio data of the combined video along with the conference ID and the recording ID to the information processing system50.

S65: The communication unit51of the information processing system50receives the request to convert the audio data. The text conversion unit56converts the audio data into text data using the speech recognition service system80. The communication unit51stores the text data in the same storage location as the storage location of the combined video. In the record information storage area5002, the text data is association with the combined video by the conference ID and the recording ID. In another example, the communication terminal10requests the speech recognition service system80to perform speech recognition and stores text data received from the speech recognition service system80in the storage location.

S66: The upload unit20implemented by the information recording application41stores the combined video in the storage location of the combined video via the communication unit11. In the record information storage area5002, the combined video is associated with the conference ID and the recording ID. For the combined video, “Uploaded” is recorded.

S67: The communication unit51of the information processing system50associates the object information transmitted from the electronic whiteboard2during the conference with the conference ID, and stores the object information in the same storage location as the storage location of the combined video. Therefore, the object information, the combined video, and the text data are associated with each other by the conference ID.

Since the user is notified of the storage location, the user can share the combined video with other participants by sending the storage location via e-mail or the like. Even when the combined video, the audio data, the text data, and the object information are generated by different devices or apparatuses, the video and data are collectively stored in one storage location. Thus, the user can view the data later in a simple manner.

According to the present embodiment, when the user ends the recording (ends the conference), the information processing system50deletes the association between the conference ID and the device ID (releases the electronic whiteboard2). Further, the electronic whiteboard2displays the end of the recording and resumes displaying or outputting the device ID.

In the present embodiment, processing at the end of recording will be described similar to Embodiment 3, but a different ending method will be described.

Note that the present embodiment is described on an assumption that the hardware configurations ofFIGS.4,5, and8described in the above embodiment are applicable.

Functions

FIG.41is an example of a functional block diagram illustrating, as individual blocks, functions of the communication terminal10, the meeting device60, and the information processing system50of the record creation system100. In the following description with reference toFIG.41, the differences fromFIG.9are mainly described.

InFIG.41, the communication terminal10includes an end detection unit24. The end detection unit24stores the shape of the electronic whiteboard2and determines whether the electronic whiteboard2is detected in the panoramic image. First, when a two-dimensional code or a barcode is detected, the end detection unit24detects the shape (circumscribed rectangle) of the electronic whiteboard2in the conference room using pattern matching or machine learning and stores the detected shape.

FIG.42illustrates an image241representing the shape of the electronic whiteboard2detected from the panoramic image. During the recording, the end detection unit24repeatedly detects the electronic whiteboard2from the panoramic image using the image241inFIG.42stored as a reference pattern. The end detection unit24determines that the conference (recording) has ended based on a determination of absence of the electronic whiteboard2in the panoramic image (surrounding image data).

According to such processing, the user can end the conference without pressing the stop recording button227. The user can end the conference with a gesture of blocking the electronic whiteboard2from the meeting device60with his/her hand (or by operating a mute button). Alternatively, the user can end the conference by leaving the conference room with the communication terminal10and the meeting device60connected to each other.

In addition, the end detection unit24may detect pulling out of a USB cable (wired cable) from the communication terminal10. to detect the end of the conference. The device communication unit16detects that the USB cable has been pulled out, for example, when the external device I/F508detects no voltage and notifies the end detection unit24of the detection. In another example, the device communication unit16detects communication interruption, for example, on the basis of no response from the meeting device60. Also in this case, the user can end the conference by pulling out the cable, which is normally performed, without pressing the stop recording button227.

Operation or Processing

FIG.43is a sequence diagram illustrating processing performed by the record creation system100when ending the recording of a conference. In the following description with reference toFIG.43, the differences fromFIG.39are mainly described.

S70: The terminal communication unit61of the meeting device60transmits the panoramic image to the communication terminal10.

S71: The end detection unit24detects that the electronic whiteboard2is not detected from the panoramic image as described above or detects that the USB cable is unplugged. Note that the meeting device60can also detect that the electronic whiteboard2is not detected from the panoramic image.

Subsequent processing may be similar to the processing inFIG.39.

According to the present embodiment, in addition to the effect of Embodiment3, it is possible to reduce the number of operation steps of the user for ending the conference (recording).

In the present embodiment, processing at the end of recording will be described similar to Embodiment 3, but a different ending method will be described.

Note that the present embodiment is described on an assumption that the hardware configurations ofFIGS.4,5, and8described in the above embodiment are applicable.

Functions

FIG.44is an example of a functional block diagram illustrating, as individual blocks, functions of the communication terminal10, the meeting device60, and the information processing system50of the record creation system100. In the following description with reference toFIG.44, the differences fromFIG.33are mainly described.

InFIG.44, the communication terminal10includes the end detection unit24. Based on the determination that a pilot signal is not included in the sound received by the microphone of the meeting device60, for example, in a predetermined period set by a manufacturer or provider of the system, the end detection unit24detects that the conference has ended. That is, during the conference (during recording), the electronic whiteboard2intermittently outputs the pilot signal, and the conference end (recording end) can be detected when the meeting device60no longer receives the pilot signal.

According to such processing, even if the user does not press the stop recording button227, the user can end the conference with a gesture of blocking the electronic whiteboard2from the meeting device60with his/her hand or by leaving the conference room with the communication terminal10and the meeting device60connected to each other.

Operation or Processing

FIG.45is a sequence diagram illustrating processing performed by the record creation system100when ending the recording of a conference. In the following description with reference toFIG.45, the differences fromFIG.39are mainly described.

S80: The terminal communication unit61of the meeting device60transmits sound data to the communication terminal10.

S81: The end detection unit24determines whether or not a pilot signal is included in the sound data transmitted from the meeting device60as described above. The information recording application41may directly perform this determination on the sound collected by the information recording application41, or may receive the result indicating that the pilot signal is not detected, from the meeting device60.

Subsequent processing may be similar to the processing inFIG.39.

According to the present embodiment, in addition to the effect of Embodiment 3, it is possible to reduce the number of operation steps of the user for ending the conference (recording).

While the present disclosure has been described above using the embodiment, the embodiment does not limit the present disclosure in any way. Various modifications and replacements may be made within a scope not departing from the gist of the present disclosure. For example, elements and/or features of different illustrative embodiments may be combined with each other and/or substituted for each other within the scope of the present invention. Any one of the above-described operations may be performed in various other ways, for example, in an order different from the one described above.

For example, the communication terminal10and the meeting device60may be integral with each other. In one example, the meeting device60is externally attached to the communication terminal10. The meeting device60may be implemented by a hemispherical camera, a microphone, and a speaker connected to one another by cables.

The meeting device60may be disposed at the second site101. The meeting device60at the second site101separately creates a combined video and text data. A plurality of meeting devices60may be disposed at a single site. In this case, multiple records are created for each meeting device60.

The arrangement of the panoramic image203, the talker images204, and the screen of the application in the combined video in the present embodiment is merely an example. The panoramic image203may be displayed below the talker images204, the user may change the arrangement, or the user may switch between non-display and display individually for the panoramic image203and the talker images204during playback.

The functional configurations illustrated in, for example,FIG.9are divided according to main functions in order to facilitate understanding of processing executed by the communication terminal10, the meeting device60, and the information processing system50. No limitation is intended by how the functions are divided by process or by the name of the functions. The processes performed by the communication terminal10, the meeting device60, and the information processing system50may be divided into a greater number of processing units in accordance with the content of the processing. In addition, a single processing unit can be further divided into a plurality of processing units.

The apparatuses or devices described in one embodiment are just one example of multiple computing environments that implement the one embodiment in this specification. In some embodiments, the information processing system50includes multiple computing devices, such as a server cluster. The plural computing devices communicate with one another through any type of communication link including a network, shared memory, etc., and perform the processes disclosed herein.

The information processing system50may share the processing steps disclosed herein, for example, steps inFIG.20or the like in various combinations. For example, a process performed by a predetermined unit may be performed by a plurality of information processing apparatuses included in the information processing system50. Further, the elements of the information processing system50may be combined into one server apparatus or are allocated to multiple apparatuses.

Each of the functions of the above-described embodiments may be implemented by one or more pieces of processing circuitry. The term “processing circuit or circuitry” used herein refers to a processor that is programmed to carry out each function by software such as a processor implemented by an electronic circuit, or a device such as an application specific integrated circuit (ASIC), digital signal processor (DSP), field programmable gate array (FPGA), or existing circuit module that is designed to carry out each function described above.

Processors are considered processing circuitry or circuitry as they include transistors and other circuitry therein. In the disclosure, the circuitry, units, or means are hardware that carries out or are programmed to perform the recited functionality. The hardware may be any hardware disclosed herein or otherwise known which is programmed or configured to carry out the recited functionality. When the hardware is a processor which may be considered a type of circuitry, the circuitry, means, or units are a combination of hardware and software, the software being used to configure the hardware and/or processor.