Abstract:
A system for resolving disputes in an online answers community is disclosed. The system improves the system&#39;s ability to resolve reports of abuse.

Description:
FIELD OF THE INVENTION 
     The present invention relates to a system for resolving disputes, including but not limited to within an online answers community. More particularly, the system improves the (content provider&#39;s existing) system&#39;s ability to resolve reports of disputes. 
     BACKGROUND 
     Content providers sometimes provide online “answers” communities where people can seek information and opinions on various topics. Within an answers community, users understand that they rely upon information obtained therein at their own risk. For such a community to be effective, it is important to screen out bad behavior such as flaming, spamming, abusive language, and pornographic references. Such behavior can result in abuse reports. 
     There can be a large number of users in such a community, thereby giving rise to a high volume of user activity therein. High volumes of user activity can give rise to high volumes of abuse reports. However, abuse reports must also be verified for accuracy, so as to prevent an innocent candidate from being needlessly reprimanded or banned from the community. 
     One way to properly manage abuse reports is for all abuse reports to be handled by a customer care division of the content provider. A customer care division can investigate each report and decides what action should be taken for the report content. However, such investigations and actions can be time-consuming. Consequently, a mechanism for resolving disputes in an online community, including but not limited to an answers community, is desired. 
     The approaches described in this section are approaches that could be pursued, but not necessarily approaches that have been previously conceived or pursued. Therefore, unless otherwise indicated, it should not be assumed that any of the approaches described in this section qualify as prior art merely by virtue of their inclusion in this section. 
    
    
     
       BRIEF DESCRIPTION OF THE DRAWINGS 
       The present invention is illustrated by way of example, and not by way of limitation, in the figures of the accompanying drawings and in which like reference numerals refer to similar elements and in which: 
         FIG. 1  is a block diagram that illustrates an example system for managing appeals in an online community, according to an embodiment of the invention; 
         FIG. 2  is a flowchart showing an example use of the system, according to an embodiment of the invention; and 
         FIG. 3  shows a computer system upon which embodiments of the invention may be implemented. 
     
    
    
     DETAILED DESCRIPTION 
     In the following description, for the purposes of explanation, numerous specific details are set forth in order to provide a thorough understanding of the present invention. It will be apparent, however, that the present invention may be practiced without these specific details. In other instances, well-known structures and devices are shown in block diagram form in order to avoid unnecessarily obscuring the present invention. 
     General Overview 
     A content provider employs a system for improving the process of dealing with abuse reports within a community, such as an online answers community. The system provides a community-based jury mechanism which resolves some of the abuse reports, without needing to engage the staff of the content provider. 
     Explanation of System 
     There can be numerous types of undesirable content within an online community, such as an online answers community. One type of undesirable content might be spam, while another type might be content that is offensive, defamatory, or pornographic. To resolve these and other issues,  FIG. 1  shows a system  100  for resolving abuse reports within an online community. One type of online community could be an online answers community, although the system  100  could also be utilized within other types of online communities. The system  100  works within any community of users that can upload viewable content which can be viewable by other members, including not just answers but also for example online gaming. 
     The system  100  tracks abuse reports  104  from various users. The system  100  also can automatically sense various forms of unwanted content, such as photographs, specific text terms, and audio files. A report module  108  initially receives the abuse reports  104 . 
       FIG. 1  also shows a reputation module  112  connected to a scoring module  116 , and a customer care module  120  connected to a jury module  124 . The reputation module assists in measuring the reputations of users, and works in conjunction with the scoring module  116 . The jury module  124  uses the information from the reputation module  112  to assist users that wish to become jurors. Abuse reports that are too close to decide can be referred to the customer care module  120 , in which a staff of individuals review the abuse reports. 
     The reputation module  112  is stored within the report module  108 , which handles abuse reports. The reputation module  112  takes information from the scoring module  116  which tracks the activities and other data regarding the various users. To make an initial decision as to whether to forward the issue onto the next level, the reputation module  112  calculates a reputation score of both a “plaintiff” (the user which reported the potential abuse) and the “defendant” (the user which is accused of inputting the potential abuse). This reputation score includes information such as how often a user logs on, how reliable that user&#39;s input has been in the past, and whether other abuse reports have been filed against that user. 
     Any decision made by the reputation module  112  includes some factoring of the reputation scores of both plaintiff and defendant. For example, the higher the reputation score of the plaintiff, or the lower the reputation of the defendant, the more likely is it that the decision made is adverse to the defendant. Assuming a decision adverse to the defendant is produced by the reputation module  112 , a defendant can still appeal the result from the reputation module  112 . Appeals are passed onto the jury module  124  for resolution. If the jury module  124  cannot decide, the appeal is then referred to the customer care division  120  which, as stated previously, is staffed by the content provider. Reducing the number of appeals that customer care division  120  must review can reduce labor costs and time. The system  100  achieves reductions by performing community-based screening using the jury module  124 , which may reduce a substantial portion of appeals. 
     Jury Module 
     The jury module  124  selects “jurors” for resolving the appeal. As part of the selection process, the jury module  124  uses the reputation module  112  to screen out users with a below-average reputation, users whose rate of login is below a certain threshold, and users who have served on a jury recently. 
     A user&#39;s reputation as a potential juror can be measured by a variety of means, including but not limited to the amount of interaction of that user with the others in the community, how often a user logs in (“login rate”), and feedback provided by other users. To measure a reputation, the scoring module  116  uses a machine-learning based evaluation of the content that the user contributed to the system  100 . Such an evaluation includes determining whether the questions/answers contributed by that user are spam or not. Other ways of measuring a user&#39;s reputation can include tallying responses to the question “did this user&#39;s input help you”, “did this review help you”, or other similar inquiry. Additional ways of measuring a user&#39;s reputation can include how much agreement that user had with majority (or customer service) decisions in the previous jury participations. 
     Within the jury module  124 , the threshold of the login rate to determine eligibility to serve as a juror is adjustable. A higher threshold of login-rate ensures only choosing users likely to see the jury communications within a very short time interval, thus ensuring that the appeal be resolved by the jury module  124  within a shorter time. However, a higher threshold also reduces the amount of possible people who could be chosen as jurors. The actual thresholds can be determined by tracking the number of juror appeals required, the distribution of login-rate among users, and a desired goal of time to resolve each appeal. 
     The jury module  124  will also screen users to ensure that users are considered only for those appeals which involve defendants that the users have no relationship with, as decided according to the implicit user-user graph considering all action in the system such as voting and answering. The system  100  can choose jurors using, for example, a random sampling from users who have never interacted with any of the parties in dispute and who have the required login rate. 
     Selection of Jurors 
     Potential jurors can be selected as follows. Appeals to be resolved will be held in a common pool of pending items. Every time a user logs in, the system  100  will check if s/he is eligible for serving as a juror for one of the appeals within the pool. If eligible, the system  100  will present that potential juror a subset of those appeals. 
     The subset of appeals to be presented to the potential juror means if there were, for example, one hundred issues pending to be resolved, for each possible juror, the system  100  could be set up to only display a predetermined threshold of appeals, for example two. The exact numbers of course depend on the parameters of the system, which can be adjusted and tuned according to desired performance. To participate and become a juror, a user must opt-in using for example an onscreen check-box or other user-selectable indicator, and then go on to read the appeal and the guidelines. 
     Example of Operation 
       FIG. 2  is a flowchart demonstrating an example usage of the system  100 . At step  204 , a user Q reports a user P&#39;s content as abusive. At step  208 , the reputation system  112  makes an initial judgment whether or not P&#39;s content truly represents a potential abuse. If P&#39;s content is determined to be potential abuse, at step  212  the system  100  hides the content from other users, and then notifies user P. 
     At step  216 , user P appeals the determination. At step  220 , a jury is chosen. At step  224 , the appeal is submitted to a set of jurors chosen from the community. At step  228 , a jury determination is made. If the jury decision is conclusive (for example &gt;=80% agreement), the content is either made visible or kept hidden. If the jury decision is undecided (for example &lt;=80% agreement), the determination is sent to the customer care division  120 . 
     The size of a jury can be related to the reputations of the plaintiff and defendant. Where the respective reputations are close, a larger jury can be used in order to get a more accurate sampling. One potential threshold can be to have at least three members who actually provide input to the jury decision, although other thresholds can also be used. The system  100  strives to stay between unanimous and simple majority, to for example keep a safe margin for errors, for instance 75%. 
     The system  100  is structured to achieve turnaround within a single day, where possible. Each time the jurors login into the system, they are shown a box with the remaining time to respond. Whether they receive emails or IMs could depend on the initial user agreements that they sign. 
     An agreement among the verdicts of the jurors can be used to prioritize abuse reports for attention by the customer care module  120 , which may be staffed by human reviewers. Verdicts with strong agreement can be considered final and not forwarded to the customer care division  120 , or granted a lower priority therein. Meanwhile, verdicts with is no strong consensus can receive higher priority from the customer care division  120 . 
     Using the system  100 , the online answers community  102  can self-moderate abuses. Also, the content provider can achieve reduced complaints, yet also reduce its own work-load. 
     User-User Graph 
     As a way of illustrating the juror selection process, suppose that users A, B, C and D are being considered for use as potential jurors. Now suppose A answered a question that has been put forward by B. Now suppose C rates that answer, and at the same time answers a question from D. A user-user graph could be formed from this data having various edges or data points. Such a juror-juror graph would look like the following. The edges (salient data points) in this graph would be (A,B), (C, A), (C,B), and (C,D). 
     The specific categories of actions which are deemed as interactions are also adjustable. For example, answering a question indicates more interaction than rating a response. Accordingly, if only question-answering is considered to be an action, then the only edges will be (A, B) and (C, D). 
     Users that opt-in can be rewarded with points or other types of reward mechanisms. Thus, honestly reporting their evaluations will be a good strategy. This in turn leads to users being highlighted in various community boards displayed on the websites, e.g. “top  100  reviewers”, “top reviewer of the week”, “experts of this year”, or other designations. Such public recognition can be incentive for users to act as jurors. It is also possible to allow cooperative jurors to use the points accumulated as a currency across various other product offerings of the content provider. 
     Hardware Overview 
       FIG. 3  is a block diagram that illustrates a computer system  300  upon which an embodiment of the invention may be implemented. Computer system  300  includes a bus  302  or other communication mechanism for communicating information, and a processor  304  coupled with bus  302  for processing information. Computer system  300  also includes a main memory  306 , such as a random access memory (RAM) or other dynamic storage device, coupled to bus  302  for storing information and instructions to be executed by processor  304 . Main memory  306  also may be used for storing temporary variables or other intermediate information during execution of instructions to be executed by processor  304 . Computer system  300  further includes a read only memory (ROM)  308  or other static storage device coupled to bus  302  for storing static information and instructions for processor  304 . A storage device  310 , such as a magnetic disk or optical disk, is provided and coupled to bus  302  for storing information and instructions. 
     Computer system  300  may be coupled via bus  302  to a display  312 , such as a cathode ray tube (CRT), for displaying information to a computer user. An input device  314 , including alphanumeric and other keys, is coupled to bus  302  for communicating information and command selections to processor  304 . Another type of user input device is cursor control  316 , such as a mouse, a trackball, or cursor direction keys for communicating direction information and command selections to processor  304  and for controlling cursor movement on display  312 . This input device typically has two degrees of freedom in two axes, a first axis (e.g., x) and a second axis (e.g., y), that allows the device to specify positions in a plane. 
     The invention is related to the use of computer system  300  for implementing the techniques described herein. According to one embodiment of the invention, those techniques are performed by computer system  300  in response to processor  304  executing one or more sequences of one or more instructions contained in main memory  306 . Such instructions may be read into main memory  306  from another computer-readable medium, such as storage device  310 . Execution of the sequences of instructions contained in main memory  306  causes processor  304  to perform the process steps described herein. In alternative embodiments, hard-wired circuitry may be used in place of or in combination with software instructions to implement the invention. Thus, embodiments of the invention are not limited to any specific combination of hardware circuitry and software. 
     The term “computer-readable medium” as used herein refers to any medium that participates in providing data that causes a computer to operation in a specific fashion. In an embodiment implemented using computer system  300 , various computer-readable media are involved, for example, in providing instructions to processor  304  for execution. Such a medium may take many forms, including but not limited to storage media and transmission media. Storage media includes both non-volatile media and volatile media. Non-volatile media includes, for example, optical or magnetic disks, such as storage device  310 . Volatile media includes dynamic memory, such as main memory  306 . Transmission media includes coaxial cables, copper wire and fiber optics, including the wires that comprise bus  302 . Transmission media can also take the form of acoustic or light waves, such as those generated during radio-wave and infra-red data communications. All such media must be tangible to enable the instructions carried by the media to be detected by a physical mechanism that reads the instructions into a computer. 
     Common forms of computer-readable media include, for example, a floppy disk, a flexible disk, hard disk, magnetic tape, or any other magnetic medium, a CD-ROM, any other optical medium, punchcards, papertape, any other physical medium with patterns of holes, a RAM, a PROM, and EPROM, a FLASH-EPROM, any other memory chip or cartridge, a carrier wave as described hereinafter, or any other medium from which a computer can read. 
     Various forms of computer-readable media may be involved in carrying one or more sequences of one or more instructions to processor  304  for execution. For example, the instructions may initially be carried on a magnetic disk of a remote computer. The remote computer can load the instructions into its dynamic memory and send the instructions over a telephone line using a modem. A modem local to computer system  300  can receive the data on the telephone line and use an infra-red transmitter to convert the data to an infra-red signal. An infra-red detector can receive the data carried in the infra-red signal and appropriate circuitry can place the data on bus  302 . Bus  302  carries the data to main memory  306 , from which processor  304  retrieves and executes the instructions. The instructions received by main memory  306  may optionally be stored on storage device  310  either before or after execution by processor  304 . 
     Computer system  300  also includes a communication interface  318  coupled to bus  302 . Communication interface  318  provides a two-way data communication coupling to a network link  320  that is connected to a local network  322 . For example, communication interface  318  may be an integrated services digital network (ISDN) card or a modem to provide a data communication connection to a corresponding type of telephone line. As another example, communication interface  318  may be a local area network (LAN) card to provide a data communication connection to a compatible LAN. Wireless links may also be implemented. In any such implementation, communication interface  318  sends and receives electrical, electromagnetic or optical signals that carry digital data streams representing various types of information. 
     Network link  320  typically provides data communication through one or more networks to other data devices. For example, network link  320  may provide a connection through local network  322  to a host computer  324  or to data equipment operated by an Internet Service Provider (ISP)  326 . ISP  326  in turn provides data communication services through the world wide packet data communication network now commonly referred to as the “Internet”  328 . Local network  322  and Internet  328  both use electrical, electromagnetic or optical signals that carry digital data streams. The signals through the various networks and the signals on network link  320  and through communication interface  318 , which carry the digital data to and from computer system  300 , are exemplary forms of carrier waves transporting the information. 
     Computer system  300  can send messages and receive data, including program code, through the network(s), network link  320  and communication interface  318 . In the Internet example, a server  330  might transmit a requested code for an application program through Internet  328 , ISP  326 , local network  322  and communication interface  318 . 
     The received code may be executed by processor  304  as it is received, and/or stored in storage device  310 , or other non-volatile storage for later execution. In this manner, computer system  300  may obtain application code in the form of a carrier wave. 
     In the foregoing specification, embodiments of the invention have been described with reference to numerous specific details that may vary from implementation to implementation. Thus, the sole and exclusive indicator of what is the invention, and is intended by the applicants to be the invention, is the set of claims that issue from this application, in the specific form in which such claims issue, including any subsequent correction. Any definitions expressly set forth herein for terms contained in such claims shall govern the meaning of such terms as used in the claims. Hence, no limitation, element, property, feature, advantage or attribute that is not expressly recited in a claim should limit the scope of such claim in any way. The specification and drawings are, accordingly, to be regarded in an illustrative rather than a restrictive sense.