Patent ID: 12205564

The same or similar reference signs in the drawings represent the same or similar components.

DETAILED DESCRIPTION OF ILLUSTRATIVE EMBODIMENTS

The present invention will be further described in detail in combination with the drawings.

In a typical configuration of the present application, a terminal, a device serving the network and a trusted party all comprise one or more processors (CPU), input/output interfaces, network interfaces and memory.

The memory may include non-permanent memory, Random Access Memory (RAM) and/or a non-volatile memory in computer-readable media, such as Read-Only Memory (ROM) or flash RAM. The memory is an example of a computer-readable medium.

The computer-readable medium includes permanent and non-permanent, removable and non-removable media, and information storage can be implemented by any method or technology. The information can be computer-readable instructions, data structures, program modules, or other data. Examples of computer storage medium include, but are not limited to, Phase-Change RAM (PRAM), Static Random Access Memory (SRAM), Dynamic Random Access Memory (DRAM), other types of Random Access Memory (RAM), Read-Only Memory (ROM), Electrically Erasable Programmable Read-Only Memory (EEPROM), flash memory or other memory technologies, CD-ROM, Digital Versatile Disc (DVD) or other optical storage, Magnetic cassettes, magnetic tape disk storage or other magnetic storage devices or any other non-transmission media, which can be used to store information that can be accessed by computing devices. According to the definition in this article, computer-readable medium does not include non-transitory computer-readable media (transitory media), such as modulated data signals and carrier waves.

In order to further elaborate the technical means adopted in the present application and the obtained effect, the technical scheme of the present application is described clearly and completely in combination with the attached drawings and the better embodiment.

FIG.1shows a flowchart of a method for displaying music score in target music video according to an aspect of the present application, the method is executed by a device1, and the method includes the following steps;S11determining a digital music score of music corresponding to a target music video;S12determining a digital music score segment matched with a current playing progress of the target music video based on a playing progress of the target music video and the digital music score;S13rendering the digital music score segment to generate a corresponding music score segment image based on a preset condition;S14presenting the music score segment image on a playing interface corresponding to the target music video.

In the present application, the method is executed by the device1, and the device1includes at least any one of a user terminal device or a network device. Specifically, the above steps may be executed by the user terminal device, or some steps may be executed by the user terminal device and some steps may be executed by the network device, so as to be implemented by the cooperation of the user device and the network device, which is not limited in the present solution. Wherein, when the device1is a network device, the device1includes, but is not limited to, a computing device and/or cloud. The computing device includes, but is not limited to, a personal computer, a notebook computer, an industrial computer, a network host, a single network server, and multiple network server sets; the cloud is composed of a large number of computers or network servers based on Cloud Computing, wherein, the Cloud Computing is a kind of distributed computing, which is a virtual supercomputer composed of a group of loosely coupled computer sets.

When the device1is a user terminal device, the device1includes but is not limited to various intelligent terminals, such as smart phones, mobile intelligent devices, personal computers, etc. Herein, the computing device and/or the cloud are only examples, and other existing or possible future device and/or resource sharing platforms, if applicable to the present application, shall also be included in the protection scope of the present application, and are included here by reference.

In the embodiment, in the step S11, the digital music score of the music corresponding to the target music video is determined. Wherein, the target music video includes video contents and corresponding music contents. In the step, the digital music score corresponding to the music contents needs to be determined. Specifically, the user can search for corresponding digital music score according to the music, and upload the searched digital music score to the device1.

Preferably, wherein, the step S11includes: searching in a digital music score database based on the target music video to determine the digital music score corresponding to the target music video.

In the embodiment, a digital score database can be established in advance, and the digital score database can be saved according to a music name or an audio track, and then search in the digital score database according to the music name or the audio track corresponding to the target music video to determine the digital score corresponding to the target music video. Herein, the search method is only an example and is not specifically limited.

Preferably, wherein, the establishment of the digital music score database includes: separating a corresponding audio from an obtained music video and separating different audio tracks from the audio; generating the digital music score corresponding to the audio based on the different audio tracks to establish the digital music score database. Through the establishment of the digital music score database, it is convenient to search the corresponding digital music score.

Specifically, a large number of music videos can be obtained in advance and the corresponding audios can be separated from the obtained music videos, and then different audio tracks can be separated from the corresponding audios. Then, the digital music scores corresponding to the audios are generated and saved based on the different audio tracks, so as to establish the digital music score database. Specifically, the audio track can be converted into MIDI (Musical Instrument Digital Interface), and information such as sound frequency, amplitude, and waveform in an MIDI file can be analyzed to generate a corresponding digital score database.

In addition, the user can also upload a corresponding digital score to create or update the digital score database. The above method of establishing a digital music score database is only an example, and other existing or possible future methods, if applicable to the present application, shall also be included in the protection scope of the present application, and are included herein by reference.

Continuing in the embodiment, in the step S12, the device1determines the digital music score segment matched with the current playing progress of the target music video based on the playing progress of the target music video and the digital music score.

Specifically, the digital music score segment can be marked with time units, and then playing progress time information of the target music video can be corresponded to time information of the digital music score to determine a time point of the digital music score corresponding to a time point of the current playing progress, and digital score between the time point of the current playing progress and a time point of a next preset time interval is determined as the digital music score segment matched with the current playing progress of the target music video. Through the way of time marking, the corresponding digital music score segment can be matched quickly, and the matching efficiency is improved.

Preferably, wherein, the method further includes: S15(not shown) analyzing the digital music score to determine a corresponding digital music score data structure; wherein, the step S12includes:determining the digital music score segment matched with the playing progress of the target music video based on the playing progress of the target music video and the digital music score data structure corresponding to the digital music score.

In the embodiment, data structure information of the digital music score can be determined through the analysis of the digital music score. The data structure information is configured to accurately record the digital music score, and the data structure information corresponds to the whole music process, so that the playing progress of the target music video and the digital music score data structure corresponding to the digital music score can be matched to determine a data structure part corresponding to the current playing progress. Then, the data structure part corresponding to the current playing progress and time structure part of a next preset segment are regarded as the digital music score segment matching the playing progress of the target music video.

Herein, the above method of determining the digital music score segment matching the current playing progress of the target music video is only an example, and other existing or possible methods in the future, if applicable to the application, should also be included in the protection scope of the application, and are included herein by reference.

Preferably, wherein, the method further includes: S16(not shown) adjusting the digital music score data structure when the digital music score segment does not match the playing progress of the target music video.

In the embodiment, if the digital music score segment cannot match the playing progress of the target music video, the digital music score data structure can be adjusted accordingly to make the digital music score segment match the playing progress of the target music video.

Continuing in the embodiment, in the step S13, rendering the digital music score segment to generate the corresponding music score segment image based on the preset condition.

For example, MusicXML is a file format developed by W3C for recording music scores. Based on XML, many programs can render music scores with it.FIG.2shows a preferred rendering example of the present application.

Specifically, in order to make the user have a better experience, the digital music score segment will be rendered to generate the corresponding music score segment image. Specifically, for the same digital music score segment, it can be rendered into different forms of music score segment images such as staff score, numbered musical notation, musical instrument fingering, etc. The specific rendered image can be limited based on the preset condition. Herein, the music score segment image is only an example, and other graphical forms that can be applied to the present application are also included in the application. As shown inFIG.3(a),FIG.3(b)andFIG.3(c), respectively show the music score segment images corresponding to the staff score, the numbered musical notation, and piano fingering.

Preferably, wherein, the preset condition includes any one of the following: a current selection operation on a type of the music segment image by a user; a historical selection operation on a type of the music segment image by a user; a musical instrument corresponding to the target music video.

Wherein, when the preset condition is the current selection operation of the type of the music score segment image by the user, the user can select a desired type of the music score segment image by himself. For example, the device1may provide a selection button for the user to select. Wherein, the types of the music score segment images include, but are not limited to, different types such as staff score, numbered notation, guitar notation, and musical instrument fingering.

When the preset condition is the historical selection operation of the type of the musical score segment image by the user, the device1can automatically determine the type for the user according to the historical selection operation by the user. For example the type frequently selected by the user can be determined as the current type of the musical score segment image, or, the type last selected is taken as the current type of the music score segment, etc.

When the preset condition is the musical instrument corresponding to the target music video, the method further includes determining the musical instrument corresponding to the music in the target music video, and rendering the digital music score segment to generate a music score segment image corresponding to the musical instrument. For example, the fingering of the musical instrument is provided to the user as a music score segment image. For example, if it is determined that the musical instrument is a piano, the piano fingering segment image may be provided to the user.

Continuing in the embodiment, in the step S14, presenting the music score segment image on a playing interface corresponding to the target music video. Specifically, the music score segment image may be covered over the target music video to be presented to the user.

Preferably, wherein, the step S14includes: presenting the music score segment image on the playing interface corresponding to the target music video in a form of the music score segment image moving relative to at least one frame of the target music video. Specifically, the music score segment image can be presented on the playing interface corresponding to the target music video in a form similar to text comments, and the music score segment image changes with time.

In a preferred embodiment, referring toFIG.4, wherein, steps S1′, S2′, S3′, and S4′ inFIG.4are the same as or basically the same as the steps S11, S12, S13, and S14in the embodiment ofFIG.1. Therefore, it will not be repeated here, and it is only included herein by reference. Preferably, the step S5′ includes: adjusting a presentation form of the music score segment image on the playing interface corresponding to the target music video based on an adjustment operation on the music score segment image by a user.

In the embodiment, the user can adjust the music score segment image presented on the interface according to his own needs, so that a presentation form corresponding to the music score segment image is more in line with the user's experience needs. Specifically, adjustment buttons and adjustment mode prompt information can be set in the interface, and the corresponding adjustment operations can be implemented according to the corresponding adjustment mode prompt information by the user.

Preferably, wherein the adjustment operation includes but is not limited to at least any one of the following: a color adjustment operation; a position adjustment operation; a size adjustment operation; a shape adjustment operation; an opacity adjustment operation. In the embodiment, the user can adjust the color, position, size, shape, or image opacity of the music score segment image. Specifically, the corresponding adjustment operation can be implemented according to the buttons corresponding to each different adjustment operation. Herein, the present application does not limit the specific way of how to implement the adjustment operation.

FIG.5shows a flowchart of a method for displaying music score in target music video according to another preferred embodiment of the present application. In the embodiment, the corresponding audio file is extracted from the video file, and the different audio tracks are separated from the audio file, such as audio track1, audio track2, etc., and then the music score data structure is obtained by converting these audio tracks into MIDI, or the music score data structure can be obtained by digitizing the music score. Wherein the music score data structure can be adjusted by a viewer, then the corresponding music score segment image is rendered based on the music score data structure, thus the corresponding music score segment image can be superimposed on a player, so that the player can present the corresponding music score segment image when playing the video file.

Compared with the prior art, the present application determines the digital music score of the music corresponding to the target music video, and determines the digital score segment matched with the current playing progress of the target music video based on the playing progress of the target music video and the digital music score, and then, based on the preset condition, renders the digital music score segment to generate the corresponding music score segment image, and presents the music score segment image on the playing interface corresponding to the target music video. In this way, the display of music score segments in the music video is implemented, which enables the user to obtain the corresponding music score while watching the video, increases the way for the user to learn music, and improves the user experience.

Moreover, the present application can also present the music score segment image on the playing interface corresponding to the target music video in the form of comments, and adjust the presentation form of the music score segment image on the playing interface corresponding to the target music video based on the user's adjustment operation of the music score segment image, which increases the user experience by the way of comments, moreover, the user can adjust the music score segment image, which increases the flexibility of the presentation effect.

FIG.6shows a schematic diagram of a device for displaying music score in target music video according to another aspect of the present application, the device1includes:A first device11, determining a digital music score of music corresponding to a target music video;A second device12, determining a digital music score segment matched with a current playing progress of the target music video based on a playing progress of the target music video and the digital music score;A third device13, rendering the digital music score segment to generate a corresponding music score segment image based on a preset condition;A fourth device14, presenting the music score segment image on a playing interface corresponding to the target music video.

In the present application, the method is executed by the device1, and the device1includes at least any one of a user terminal device or a network device. Specifically, the above steps may be executed by the user terminal device, or some steps may be executed by the user terminal device and some steps may be executed by the network device, so as to be implemented by the cooperation of the user device and the network device, which is not limited in the solution. Wherein, when the device1is a network device, the device1includes, but is not limited to, a computing device and/or cloud. The computing device includes, but is not limited to, a personal computer, a notebook computer, an industrial computer, a network host, a single network server, and multiple network server sets; the cloud is composed of a large number of computers or network servers based on Cloud Computing, wherein, the Cloud Computing is a kind of distributed computing, which is a virtual supercomputer composed of a group of loosely coupled computer sets.

When the device1is a user terminal device, the device1includes but is not limited to various intelligent terminals, such as smart phones, mobile intelligent devices, personal computers, etc. Herein, the computing device and/or the cloud are only examples, and other existing or possible future device and/or resource sharing platforms, if applicable to the present application, shall also be included in the protection scope of the present application, and are included here by reference.

In the embodiment, the first device of the device1is configured to determine the digital music score of the music corresponding to the target music video. Wherein, the target music video includes video contents and corresponding music content. In the step, the digital music score corresponding to the music contents needs to be determined. Specifically, the user can search for corresponding digital music score according to the music, and upload the searched digital music score to the device1.

Preferably, wherein, the first device: searching in a digital music score database based on the target music video to determine the digital music score corresponding to the target music video.

In the embodiment, a digital score database can be established in advance, and the digital score database can be saved according to a music name or an audio track, and then search in the digital score database according to the music name or the audio track corresponding to the target music video to determine the digital score corresponding to the target music video. Herein, the search method is only an example and is not specifically limited.

Preferably, wherein, the establishment of the digital music score database includes: separating a corresponding audio from an obtained music video and separating different audio tracks from the audio; generating the digital music score corresponding to the audio based on the different audio tracks to establish the digital music score database. Through the establishment of the digital music score database, it is convenient to search the corresponding digital music score.

Specifically, a large number of music videos can be obtained in advance and the corresponding audios can be separated from the obtained music videos, and then different audio tracks can be separated from the corresponding audios. Then, the digital music scores corresponding to the audios are generated and saved based on the different audio tracks, so as to establish the digital music score database. Specifically, the audio track can be converted into MIDI (Musical Instrument Digital Interface), and information such as sound frequency, amplitude, and waveform in an MIDI file can be analyzed to generate a corresponding digital score database.

In addition, the user can also upload a corresponding digital score to create or update the digital score database. The above method of establishing a digital music score database is only an example, and other existing or possible future methods, if applicable to the present application, shall also be included in the protection scope of the present application, and are included herein by reference.

Continuing in the embodiment, the second device of the device1determines the digital music score segment matched with the current playing progress of the target music video based on the playing progress of the target music video and the digital music score.

Specifically, the digital music score segment can be marked with time units, and then playing progress time information of the target music video can be corresponded to time information of the digital music score to determine a time point of the digital music score corresponding to a time point of the current playing progress, and digital score between the time point of the current playing progress and a time point of a next preset time interval is determined as the digital music score segment matched with the current playing progress of the target music video. Through the way of time marking, the corresponding digital music score segment can be matched quickly, and the matching efficiency is improved.

Preferably, wherein, the device further includes: a fifth device (not shown), analyzing the digital music score to determine a corresponding digital music score data structure; wherein, the second device:determining the digital music score segment matched with the playing progress of the target music video based on the playing progress of the target music video and the digital music score data structure corresponding to the digital music score.

In the embodiment, data structure information of the digital music score can be determined through the analysis of the digital music score. The data structure information is configured to accurately record the digital music score, and the data structure information corresponds to the whole music process, so that the playing progress of the target music video and the digital music score data structure corresponding to the digital music score can be matched to determine a data structure part corresponding to the current playing progress. Then, the data structure part corresponding to the current playing progress and time structure part of a next preset segment are regarded as the digital music score segment matching the playing progress of the target music video.

Herein, the above method of determining the digital music score segment matching the current playing progress of the target music video is only an example, and other existing or possible methods in the future, if applicable to the application, should also be included in the protection scope of the application, and are included herein by reference.

Preferably, wherein, the device further includes: a sixth device (not shown), adjusting the digital music score data structure when the digital music score segment does not match the playing progress of the target music video.

In the embodiment, if the digital music score segment cannot match the playing progress of the target music video, the digital music score data structure can be adjusted accordingly to make the digital music score segment match the playing progress of the target music video.

Continuing in the embodiment, the third device of the device1, rendering the digital music score segment to generate the corresponding music score segment image based on the preset condition.

Specifically, in order to make the user have a better experience, the digital music score segment will be rendered to generate the corresponding music score segment image. Specifically, for the same digital music score segment, it can be rendered into different forms of music score segment images such as staff score, numbered musical notation, musical instrument fingering, etc. The specific rendered image can be limited based on the preset conditions. Herein, the music score segment image is only an example, and other graphical forms that can be applied to the present application are also included in the application.

Preferably, wherein, the preset condition includes any one of the following: a current selection operation on a type of the music segment image by a user; a historical selection operation on a type of the music segment image by a user; a musical instrument corresponding to the target music video.

Wherein, when the preset condition is the current selection operation of the type of the music score segment image by the user, the user can select a desired type of the music score segment image by himself. For example, the device1may provide a selection button for the user to select. Wherein, the types of the music score segment images include, but are not limited to, different types such as staff score, numbered notation, guitar notation, and musical instrument fingering.

When the preset condition is the historical selection operation of the type of the musical score segment image by the user, the device1can automatically determine the type for the user according to the historical selection operation by the user. For example the type frequently selected by the user can be determined as the current type of the musical score segment image, or, the type last selected is taken as the current type of the music score segment, etc.

When the preset condition is the musical instrument corresponding to the target music video, the method further includes determining the musical instrument corresponding to the music in the target music video, and rendering the digital music score segment to generate a music score segment image corresponding to the musical instrument. For example, the fingering of the musical instrument is provided to the user as a music score segment image. For example, if it is determined that the musical instrument is a piano, the piano fingering segment image may be provided to the user.

Continuing in the embodiment, the fourth device of the device1, presenting the music score segment image on a playing interface corresponding to the target music video. Specifically, the music score segment image may be covered over the target music video to be presented to the user.

Preferably, wherein, the fourth device, presenting the music score segment image on the playing interface corresponding to the target music video in a form of moving across at least on frame of the target music video. Specifically, the music score segment image can be presented on the playing interface corresponding to the target music video in a form similar to a text comments, and the music score segment image changes with time.

According to another aspect of the present application, a device2there is also provided for displaying music scores in target music video, and the device2includes:A first first device, determining a digital music score of music corresponding to a target music video;A first second device, determining a digital music score segment matched with a current playing progress of the target music video based on a playing progress of the target music video and the digital music score;A first third device, rendering the digital music score segment to generate a corresponding music score segment image based on a preset condition;A first fourth device, presenting the music score segment image on a playing interface corresponding to the target music video;A first fifth device, adjusting a presentation form of the music score segment image on the playing interface corresponding to the target music video based on an adjustment operation on the music score segment image by a user.

In the embodiment, the first first device, the first second device, the first third device and the first fourth device are the same as or basically the same as the he first device, the second device, the third device and the fourth device in the embodiment ofFIG.1. It will not be repeated herein.

In the embodiment, the user can adjust the music score segment image presented on the interface according to his own needs, so that the presentation form corresponding to the music score segment image is more in line with the user's experience needs. Specifically, adjustment buttons and adjustment mode prompt information can be set in the interface, and the corresponding adjustment operations can be implemented according to the corresponding adjustment mode prompt information by the user.

Preferably, wherein the adjustment operation includes but is not limited to at least any one of the following: a color adjustment operation; a position adjustment operation; a size adjustment operation; a shape adjustment operation; an opacity adjustment operation. In the embodiment, the user can adjust the color, position, size, shape, or image opacity of the music score segment image. Specifically, the corresponding adjustment operation can be implemented according to the buttons corresponding to each different adjustment operation. Herein, the present application does not limit the specific way of how to implement the adjustment operation.

Compared with the prior art, the present application determines the digital music score of the music corresponding to the target music video, and determines the digital score segment matched with the current playing progress of the target music video based on the playing progress of the target music video and the digital music score, and then, based on the preset condition, renders the digital music score segment to generate the corresponding music score segment image, and presents the music score segment image on the playing interface corresponding to the target music video. In this way, the display of music score segments in the music video is implemented, which enables the user to obtain the corresponding music score while watching the video, increases the way for the user to learn music, and improves the user experience.

Moreover, the present application can also present the music score segment image on the playing interface corresponding to the target music video in the form of comments, and adjust the presentation form of the music score segment image on the playing interface corresponding to the target music video based on the user's adjustment operation of the music score segment image, which increases the user experience by the way of comments, moreover, the user can adjust the music score segment image, which increases the flexibility of the presentation effect.

In addition, the embodiment of the application also provides a computer-readable medium, which stores computer-readable instructions, and the computer-readable instructions can be executed by a processor to implement the above method.

The embodiment of the application also provides a device for displaying music score in target music video, wherein the device comprises:one or more processors; anda memory storing computer-readable instructions, and the computer-readable instructions, when executed, cause the processors to perform operations of the above method.

For example, the computer-readable instructions, when executed, cause the one or more processors to: determine a digital music score of music corresponding to a target music video; determine a digital music score segment matched with a current playing progress of the target music video based on a playing progress of the target music video and the digital music score; render the digital music score segment to generate a corresponding music score segment image based on a preset condition; present the music score segment image on a playing interface corresponding to the target music video.

FIG.7schematically shows a schematic diagram of a hardware architecture of a computer device2suitable for implementing the techniques described in the present application. In the embodiment, the computer device2is a device capable of automatically performing numerical calculations and/or information processing according to predefined or stored instructions, such as can be a smartphone, a tablet, a laptop, a desktop computer, a rack server, a blade server, a tower server or a rack server (including independent servers, or server clusters composed of multiple servers), etc. As shown inFIG.7, the computer device2includes, but is not limited to, a memory1110, a processor1120, a network interface1130, and a vibration element1140that can be communicated with each other through a system bus. Wherein:

The memory1110includes at least one type of computer-readable storage medium. The readable storage medium includes flash memory, hard disk, multimedia card, card type memory (such as SD or DX memory, etc.), random access memory (RAM), static random access memory (SRAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), programmable read-only memory (PROM), magnetic memory, magnetic disk, optical dis and so on. In some embodiments, the memory1110may be an internal storage module of the computer device2such as a hard disk or memory of the computer device2. In other embodiments, the memory1110may also be an external storage device of the computer device2, such as a plugged hard disk provided in the computer device2, a smart media card (SMC), secure digital (SD) card, a flash memory card, and so on. Of course, the memory1110may also include both an internal storage module and an external storage device of the computer device2. In the embodiment, the memory1110is generally used to store an operating system and various types of application software installed in the computer device2such as program codes of the game playing method based on a multimedia file and the like. In addition, the memory1110may also be used to temporarily store various types of data that have been or will be outputted.

The processor1120, in some embodiments, may be a central processing unit (CPU), a controller, a microprocessor, or other data processing chip. The processor1120is generally used to control the overall operation of the computer device2such as performing control and processing related to data interaction or communication with the computer device2. In the embodiment, the processor1120is used to run program code stored in the memory1110or process data.

The network interface1130may include a wireless network interface or a wired network interface which is generally used to establish a communication connection between the computer device2and other computer devices. For example, the network interface1130is used for connecting the computer device2to an external terminal via a network and establishing a data transmission channel and a communication connection between the computer device2and the external terminal. The network can be a wireless or wired network such as an enterprise intranet, an Internet, a Global System of Mobile communication (GSM), a Wideband Code Division Multiple Access (WCDMA), a 4G network, a 5G network, a Bluetooth, Wi-Fi, and so on.

It is to be noted thatFIG.7shows only the computer device2having components1110-1130, but it is understood that it is not required to implement all of the shown components and that more or fewer parts can be implemented in lieu thereof.

In the embodiment, the game playing method based on a multimedia file stored in the memory1110may be divided into one or more program modules and executed by one or more processors (processor1120in the embodiment) to complete the embodiment of the present application. For those skilled in the art, it is obvious that the invention is not limited to the details of the above exemplary embodiments, and can be implemented in other specific forms without departing from the spirit or basic features of the invention. Therefore, no matter from which point of view, the embodiments should be regarded as be exemplary and non restrictive. The scope of the invention is limited by the attached claims rather than the above description. Therefore, it is intended to include all changes within the meaning and scope of the equivalent elements of the claims in the invention. Any reference mark in the claims shall not be regarded as limiting the claims concerned. In addition, it is clear that the word “including” does not exclude other units or steps, and the singular does not exclude the plural. The multiple units or devices stated in the device claim can also be implemented by one unit or device through software or hardware. The first and second words are used to denote names, not any particular order.