SYSTEM AND METHOD FOR EVALUATING THE PEER REVIEW PROCESS OF SCHOLARLY JOURNALS

A computer-implemented method and system for providing a rating of the quality of a peer review process for an article is disclosed. A weighted peer review score is calculated from metadata of the article which includes information about the number of editors-in-chief, the number of associate editors, and the total number of reviewers that participated in the peer review of the article. The metadata also includes information about the number of revisions of the article. The peer review score is displayed next to a link to the article returned in the search results from the search engine. The peer review score indicates the relative thoroughness of the peer review process that the article underwent. The peer review score is included with a link to the article when the article appears in a search engine search results.

DETAILED DESCRIPTION

FIG. 1shows a system for appraising the extent to which a publication has been reviewed by means of a peer-review process such that a user conducting a search with a search engine via an internet connected electronic device can infer the quality of the publication in relation to other publications returned in the search results displayed on the electronic device.

In the peer-review process, an author20submits a first revision of an article to a publisher having one or more editors-in-chief and associate editors30(1 . . . n). The publisher30selects one or more reviewers40(1 . . . n) to review the article, and forwards the article to the reviewers40(1 . . . n). The reviewers40comment on the article and submit the comments to the publisher30. The publisher30forwards the comments, which include comments that the publisher has made, back to the author20. The author20revises the article and sends a second revision to the publisher30. The second revisions is forwarded and reviewed again, and additional revisions are created as needed until the paper is accepted by the publisher for publication. As part of this process, the publisher may also rate the reviewers40on the quality of their review and other factors.

Article revisions, comments, ratings, and the like are stored in database42by server41. The database42may store or have access to thousands of articles from thousands of journals. Server41is connected to a network10such as the internet and therefore can access other databases connected to the internet. The publisher30, reviewers40, and author20are also in communication with the network10. The network10may comprise any number and types of networks, such as the internet, a local area network (LAN), a wide area network (WAN), a virtual private network (VPN), wired networks, and wireless networks.

The database42can be searched by a user with an internet connected electronic device such as a computer50or mobile device52. The mobile device52may be, for example, a smartphone, laptop computer, iPhone, iPad, and the like.

The database42can also be searched by a search engine60. In one example, a user50accesses search engine60with a web browser, enters search terms, and the search request is transmitted to server60. Server60executes a search on database62and on other internet connected databases42and serves the results to user computer50. The results are displayed in the web browser of user computer50. The results are ranked according to the algorithm of the search engine60and include, at least, a link to an article or web page. The user can select the article or web page by clicking on the link.

One example of a search engine is Google Scholar (http://scholar.google.com/). Other examples, including examples of peer-review systems and methods are described in the U.S. patents and patent publications identified in the Background section above.

Peer Review Score (PRS) server100is also in communication with network10. Server100is a computer having well known components such as a microprocessor, memory, network interface, and a storage device such as a disk drive. The storage device stores computer executable code which when executed by the processor of server100causes the computer to carry out the methods described herein and illustrated inFIG. 2.

The server100includes a database102for storing data such as articles and scores, an article module104for receiving an article from a publisher, a weight module106for assigning weight factors of peers30and40, a metadata module for scanning the article for metadata and determining how many and what types of peers30and40reviewed the article from author20, a score module110for calculating a peer review score for the article, a display module for displaying the peer review score next to each link to each article returned in the search results from the search engine60, and an audit module113for auditing a score from the score module110. The details of the modules102-113are disclosed below.

FIG. 2shows a method carried out by the server100for appraising the extent to which a publication has been reviewed by means of a peer-review process such that a user conducting a search with a search engine via an internet connected electronic device can infer the quality of the publication in relation to other publications returned in the search results displayed on the electronic device.

An article is received208from a publisher. According to the publisher, the article has been peer-reviewed by a plurality of peers. The plurality of peers comprise at least one editor-in-chief (EIC), at least one associate editor (AE), and at least one reviewer (R). The article may be received in response to search results224from a search, or the article may be received as part of a process that monitors a publisher's database for new articles, or the article may be received because a publisher transmitted the article to the system.

The terms “transmitting” and “received” are understood herein to include the electronic transfer of a file or a link to a file over a network. If the article is not received in paper form then the article is scanned to convert it into an electronic form. If the electronic form is in an incompatible format, such as a TIFF file, then the form is converted from the incompatible format to a compatible format such a portable document format (PDF). Formats such as TIFF and PDF are merely exemplary and other file formats may be used.

In one example, the article is in a portable document format (PDF) and includes metadata. The metadata information may include, for example, the total number of peers, the total number of editors-in-chief that participated in the peer review, the total number of associate editors that participated in the peer review, the total number of reviewers that participated in the peer review, and the total number of revisions of the article. One example of a PDF format is defined International Standard ISO 19005-1:2005. Another example is ISO 32000-1. In another example, an XML file is received comprising the metadata which includes information about the peer-review process associated with the article (e.g., X, Y, Z, and V disclosed below) and bibliographic information about the article. It is appreciated that, as used herein, any reference to receiving an article equivalently includes receiving a PDF, or an XML file, or any equivalent combination or modification thereof.

A weight factor E is determined202for the editor(s)-in-chief. The weight factor E is assigned by considering the editor-in-chief s relative merit compared to that of other peers (for example AE and R) in the peer review process.

Similarly, a weight factor F is determined204for the associate editor(s). The weight factor F is assigned by considering the associate editor's relative merit compared to that of other peers (for example EIC and R) in the peer review process.

And, a weight factor G is determined206for the reviewer(s). The weight factor G is assigned by considering the reviewer's relative merit compared to that of other peers (for example EIC and AE) in the peer review process.

In one example, E, F, and G are assigned according to the responsibility and accountability level of the peer. For example, E>F>G because an editor-in-chief has the most control over whether an article is published, an associate editor has lesser control, and a reviewer has the least control. In another example, E, F, and G are assigned by considering the according to the reputations of the peer. In another example, more than one publisher participates in an online marketplace or auction to set values for E, F, and G.

In yet another example, already published articles and their peer-review histories are reviewed manually by a plurality of experts (such as EICs and AEs), scores (which will be disclosed below) are manually assigned to each article based on the review, and values of E, F, and G are determined from the dataset of articles and scores. In this example, values for E, F, and G could be determined by solving sets of linear equations for the score equation below.

In still another example, E, F, and G are constant. In a different example (e.g., branch201inFIG. 2), E, F, and G are variable and are recalculated on a periodic basis, according to the journal, according to subject or industry of the article, and the like. It is appreciated that other ways of determining/assigning E, F, and G are possible.

After the article is received at step208, the article is scanned at step209for metadata. Scanning a PDF file, an XML file, or any other equivalent file format is well understood by those skilled in the art. The metadata is scanned at step209in order to determine, for example, how many of each type of peer participated (steps210-214) in the peer review and how many revisions the article underwent (step216). In an alternative embodiment, a separate file including the metadata may be received with or instead of the article.

If the metadata is incomplete or missing, a request may be transmitted to the publisher to supply the missing metadata. The publisher replies to the request providing the missing metadata. Then, the metadata is stored with the article (for example in database102ofFIG. 1) (or separately stored when a metadata-only file is received).

At step210, a number X equal to the total number of editors-in-chief that participated in the peer review process of the article is extracted from the metadata. At step212, a number Y equal to the total number of associate editors that participated in the peer review process of the article is extracted from the metadata. At step214, a number Z equal to the total number of reviewers that participated in the peer review of the article is extracted from the metadata. And, at step216, a number V equal to the total number of revisions of the article during the peer review process is extracted from the metadata.

Next, a peer review score S is calculated at step218. S is a function of E, F, G, X, Y, Z, and V. In one example, S is not a function of a number of times the article was cited in other publications or a number of links to the article from documents or web pages accessible via the internet. For example, the score S may be calculated according to the equation:

In an alternative embodiment, the score S may be calculated according to the equation:

As mentioned above, reviewer ratings may also be received for each of the reviewers. The ratings may be obtained by scanning, at step209, the article for metadata. Alternatively, the reviewer ratings may be obtained by accessing a database (such as database92ofFIG. 1). Next, an aggregate reviewer rating R is computed from all of the reviewer ratings received. In one example, R equals the mean of all of the reviewer ratings. Then, the score S is computed where S is a function of E, F, G, X, Y, Z, V, and R. For example, the score S may alternatively be calculated according to the following equation:

In an alternative embodiment, the score S may be calculated according to the equation:

At step220, the score S is displayed next to a link to the article in the search results. In displaying the score S, the display module112ofFIG. 1transmits computer executable code which when executed in the web browser of a computer (such as user computer50or52) causes the web browser to display the score S next to the link retrieved from the search engine. The computer executable code may be transmitted226to the search engine, and the search engine may generate additional computer executable code for rendering a webpage of search results with the scores S.

FIG. 3shows an exemplary web page from a search engine with scores displayed next to a link to an article. A search316was made of the search engine for articles about “embryonic stem cells.” The results include articles for which a peer review score could not be computed, e.g., search results314, and results300,312including a peer review score308and310. The results include a link300to the article and a score308displayed next to the link300. For the article300, the peer review score (PRS)308equals 12.5. The search result312has a PRS=15 (310). The search results may include additional information such as bibliographic information301, an abstract304, and additional information and links306such as the number of citation, links to related articles, and the like.

Finally, turning back toFIG. 1, the auditing module113audits the score S from the score module110. In auditing the score, the article is transmitted to an auditor. The auditor may be, for example an independent reviewer or enforcement agency. The auditor computes a score and the auditor's score is compared with the score from the score module110. Then, for any discrepancy, S is recomputed to have the corrected value received from the auditor. In one example, the auditor returns corrected values for at least one of E, F, G, X, Y, Z, and V and the score module110recomputes the score S.

The methods and systems may be implemented on any computer communicating over any network. For example the computers may include desktop computers, tablets, handheld devices, laptops and mobile devices. The mobile devices may comprise many different types of mobile devices such as cell phones, smart phones, PDAs, portable computers, tablets, and any other type of mobile device operable to transmit and receive electronic messages.

The computer network(s) may include the internet and wireless networks such as a mobile phone network. Any reference to a “computer” is understood to include one or more computers operable to communicate with each other. Computers and devices comprise any type of computer capable of storing computer executable code and executing the computer executable code on a microprocessor, and communicating with the communication network(s). For example computer may be a web server.

References to electronic identifiers may be used which include, but are not limited to, email addresses, mobile phone numbers, user IDs for instant messaging services, user IDs for social networking application or mobile applications, user IDs and URLs for blogs and micro-blogs, URLs, bank account or financial institution numbers, routing numbers, credit and debit cards, any computer readable code, and other electronic identifiers to identify accounts, users, companies, and the like.

The systems and methods may be implemented on an Intel or Intel compatible based computer running a version of the Linux operating system or running a version of Microsoft Windows, Apple OS, and other operating systems. Computing devices based on non-Intel processors, such as ARM devices may be used. Various functions of any server, mobile device or, generally, computer may be implemented in hardware and/or in software, including in one or more signal processing and/or application specific integrated circuits.

The computers and, equivalently, mobile devices may include any and all components of a computer such as storage like memory and magnetic storage, interfaces like network interfaces, and microprocessors. For example, a computer comprises some of all of the following: a processor in communication with a memory interface (which may be included as part of the processor package) and in communication with a peripheral interface (which may also be included as part of the processor package); the memory interface is in communication via one or more buses with a memory (which may be included, in whole or in part, as part of the processor package; the peripheral interface is in communication via one or more buses with an input/output (I/O) subsystem; the I/O subsystem may include, for example, a graphic processor or subsystem in communication with a display such as an LCD display, a touch screen controller in communication with a touch sensitive flat screen display (for example, having one or more display components such as LEDs and LCDs including sub-types of LCDS such as IPS, AMOLED, S-IPS, FFS, and any other type of LCD; the I/O subsystem may include other controllers for other I/O devices such as a keyboard; the peripheral interface may be in communication with either directly or by way of the I/O subsystem with a storage controller in communication with a storage device such a hard drive, non-volatile memory, magnetic storage, optical storage, magneto-optical storage, and any other storage device capable of storing data; the peripheral interface may also be in communication via one or more buses with one or more of a location processor such as a GPS and/or radio triangulation system, a magnetometer, a motion sensor, a light sensor, a proximity sensor, a camera system, wireless communication subsystem(s), and audio subsystems.

A non-transitory computer readable medium, such as the memory and/or the storage device(s) includes/stores computer executable code which when executed by the processor of the computer causes computer to perform a series of steps, processes, or functions. The computer executable code may include, but is not limited to, operating system instructions, communication instruction, GUI (graphical user interface) instructions, sensor processing instructions, phone instructions, electronic messaging instructions, web browsing instructions, media processing instructions, GPS or navigation instructions, camera instructions, magnetometer instructions, calibration instructions, an social networking instructions. An application programming interface (API) permits the systems and methods to operate with other software platforms such as Salesforce CRM, Google Apps, Facebook, Twitter, social networking sites, desktop and server software, web applications, mobile applications, and the like. For example, an interactive messaging system could interface with CRM software and GOOGLE calendar.

A computer program product may include a non-transitory computer readable medium comprising computer readable code which when executed on the computer causes the computer to perform the methods described herein. Databases may comprise any conventional database such as an Oracle database or an SQL database. Multiple databases may be physically separate, logically separate, or combinations thereof.

The features described can be implemented in any digital electronic circuitry, with a combination of digital and analog electronic circuitry, in computer hardware, firmware, software, or in combinations thereof. The features can be implemented in a computer program product tangibly embodied in an information carrier (such as a hard drive, solid state drive, flash memory, RAM, ROM, and the like), e.g., in a machine-readable storage device or in a propagated signal, for execution by a programmable processor; and method steps can be performed by a programmable processor executing a program of instructions to perform functions and methods of the described implementations by operating on input data and generating output(s).

To provide for interaction with a user, the features can be implemented on a computer having a display device such as a CRT (cathode ray tube) or LCD (liquid crystal display) for displaying information to the user and a keyboard and a pointing device such as a mouse or a trackball by which the user can provide input to the computer. The display may be touch sensitive so the user can provide input by touching the screen.

The foregoing detailed description has discussed only a few of the many forms that this invention can take. It is intended that the foregoing detailed description be understood as an illustration of selected forms that the invention can take and not as a definition of the invention. It is only the claims, including all equivalents that are intended to define the scope of this invention.