Network graph parser

An approach for processing node data from code repository websites to generate patterns is disclosed. Node data can be parsed from a projects webpage or received from a code repository server hosting the repository website. Visualizations can be generated in a browser from the node data. The visualizations can be displayed within the browser and further be used to receive filter instructions. Refined node data can then be exported for further analysis.

TECHNICAL FIELD

Embodiments of the present disclosure relate generally to pattern detection and, more particularly, but not by way of limitation, to manipulating data via a network graph parser to expose previously undetected patterns.

BACKGROUND

A code repository website allows users to publish software code projects to the website so that other users can access, view, edit, or otherwise use the published software code. Identifying how different projects (e.g., software coding projects) are related to one another is currently impractical because the project data on the code repository websites is largely unstructured.

DETAILED DESCRIPTION

In various example embodiments, a network graph parser is implemented to parse data from websites (e.g., code repository websites) into human understandable patterns. According to some example embodiments, the code repository websites are websites or network-based publication platforms (e.g., Internet forums) that allow users to publish data viewable by other users of the website or platform. For example, a software developer can create a project page on a code repository site and publish his/her code for the project to the project page. Other uses may navigate to the project page, view, download, or modify the code for the projects.

According to some example embodiments, the network graph parser is installed as a browser plugin of an Internet browser application. A data analyst may navigate to a given page on a repository website projects, such as a page created or associated with the project or a contributor. The analyst may then trigger the parse operation by selecting a browser plugin button. The parse operation goes through the page and saves data on the page and on related pages. For example, the network graph parser may identify links to projects listed on the repository website. In some embodiments, the network graph parser may navigate to each of the projects.

The saved data may be used to generate a visual representation (e.g., a network graph) of the collected data. The data analyst may manipulate the visual representation to explore patterns. Further, the data analyst may hone down onto specific subsets by issuing filter instructions. For example, the data analyst may filter out any connections that don't have at least two connections to other nodes. Contributors may have connections to one another by working together on the same coding project, as an example. The various filter instructions expose previously invisible patterns in the network graph. The honed down data containing the pattern can then be exported over a network to a data analysis server for further analysis, according to some example embodiments.

FIG. 1is a block diagram depicting a networked system100comprising an electronic device110, and one or more components external to the electronic device110. These external components include a database system10, network120, and a plurality of repository servers130-1to130-n, that host repository websites. According to some example embodiments, the electronic device110is a client device, such as a personal computer, a tablet computer, a personal digital assistant (PDA), a mobile phone, a smart-phone, or any other web-enabled computing device with a processor and a memory. The electronic device110has installed thereon a web browser application (e.g., web browser1632inFIG. 16), on which is installed a network graph parser. According to some example embodiments, the network graph parser is integrated into the web browser application as a plugin or browser extension. Each of the plurality of repository servers130-1to130-ncomprises hardware and software. Each of the plurality of repository servers130-1to130-nis able to communicate with the electronic device110via the network120.

In some embodiments, some of the plurality of repository servers130-1to130-ncan be a part of a cloud, which can include, for example, one or more networked servers. Such networked servers may be termed a data center or a server farm. Such data centers currently are maintained by various communication network service providers. Network120can be, for example, the Internet, an intranet, a local area network, a wide area network, a campus area network, a metropolitan area network, an extranet, a private extranet, or a combination of any of these or other appropriate networks.

For the exemplary embodiment ofFIG. 1, it is understood that the electronic device110is separate from the external database system10but connected thereto by a link. Alternatively, the database system10may be disposed in an air-gapped, high-side environment, where the database system10is physically isolated from the network120and the electronic device110, such that a higher level of classified information can be maintained in the database system10.

The electronic device110may be implemented by one or more specially configured computing devices. The electronic device110may be hard-wired to perform the operations, techniques, etc. described herein. The electronic device110can include digital electronic devices such as one or more application-specific integrated circuits (ASICs) or field programmable gate arrays (FPGAs) that are persistently programmed to perform the operations, techniques, etc. described herein. The electronic device110can include one or more general purpose hardware processors (including processor circuitry) programmed to perform such features of the present disclosure pursuant to program instructions in firmware, memory, other storage, or a combination. The electronic device110can also combine custom hard-wired logic, ASICs, or FPGAs with custom programming to accomplish the methods and other features.

The electronic device110can be generally controlled and coordinated by operating system software, such as iOS, Android, Blackberry, Chrome OS, Windows XP, Windows Vista, Windows 7, Windows 8, Windows Server, Windows CE, Unix, Linux, SunOS, Solaris, VxWorks, or a proprietary operating system. The operating system controls and schedules computer processes for execution, perform memory management, provide file system, networking, I/O services, and provides a user interface functionality, such as a graphical user interface (“GUI”), among other things.

FIG. 2shows internal functional components of the network graph parser234, according to some example embodiments. In the illustrated embodiment, the network graph parser is implemented as a plug-in or browser extension for a web browser. As illustrated, the network graph parser234comprises an interface engine210, a parse engine220, a node data engine230, a visualization engine240, and an export engine250. The interface engine210is configured to interface with the browser1632as a plugin. Further, the interface engine210is configured to interface with entities outside the electronic device110, such as the repository server130. The parse engine220is configured to parse node data from a code project webpage. Node data is data of an object associated with a given project. In some example embodiments, the node data is user data (e.g., data of software developers) associated with a given project, where each user may be involved in several different software projects. In some example embodiments, the node data includes code portions (e.g., classes, functions) shared between different projects. For example, two different projects may share an Optical Character Recognition (OCR) class, and the OCR class code can be used as a node associated with each software project's network graph, as discussed in further detail below. As a further example, according to some example embodiments, metadata describing an object (e.g., code portion) may be used as a node associated with each software project's network graph.

In some example embodiments, where the repository website is configured to provide node data, the parse engine220is configured to send node data requests for the users of the repository website. The repository website can receive the requests and issue responses including the requested node data. The node data engine230is configured to process the node data received via the code projects webpage (e.g., via spidering) or received from the repository website. The node data engine230can receive filter instructions from a user and cull (e.g., refine) the node data by removing data of users that do not meet the requirements of the filter instruction, as explained in further detail below. The visualization engine240is configured to use the initial node data or the refined node data and generate different types of visualizations for display on the display screen of the electronic device110. The visualizations may include a network graph, a histogram, graphs such bar charts or data plots, and other visualizations. The export engine250is configured to export the refined dataset to an analysis server for further analysis.

FIG. 3is a flowchart representing an exemplary method300performed by an electronic device for collecting and analyzing data from repository website systems, according to some example embodiments. While the flowchart discloses the following operations in a particular order, it will be appreciated that at least some of the operations can be moved, modified, or deleted where appropriate, consistent with the teachings of the present disclosure. In the depicted embodiment ofFIG. 3, a user can utilize an electronic device (e.g., electronic device110) that comprises a web browser1632, for example, Google™ Chrome™, Mozilla™ Firefox™, Microsoft™ Internet Explorer™, etc. The web browser1632is usable to access web content (e.g. provided by the repository servers130-1to130-n) via a network (e.g., network120), such as the Internet or an intranet.

At operation310, a network graph parser234is installed as a plugin in the web browser1632of the electronic device. The network graph parser234may be termed a browser extension, according to some example embodiments. The network graph parser234extends the functionality of the web browser1632, as is described in detail below. The network graph parser234may be authored using a web technologies such as HTML, JavaScript, or CSS (Cascading Style Sheets).

Referring again toFIG. 3, at operation320, the interface engine210accesses communication and node data related content from a repository server130(e.g., one of the repository servers130-1to130-n) using the web browser1632. In the following description, reference is made generally to accessing content from a repository server130, and it will be appreciated that, unless the context indicates otherwise, such references are to accessing content from a particular repository server130, for instance the first repository server130-1.

According to some example embodiments, the repository server130is accessed through the browser1632causing sending of a request (e.g. an HTTP request) to the repository server130(in particular to a webserver included as part thereof.

Once the user has accessed the repository server130using the browser1632, they may control the browser1632to interact with the repository server130using user interface controls provided in the browser1632by the network graph parser234or using controls provided by the browser itself. In some example embodiments, the information received from the repository webservice comprises a projects webpage showing different coding or software projects associated with a user of the repository webservice.

At operation330, the parse engine220parses the data from the projects webpage and stores the data in local memory of the electronic device110. According to some example embodiments, node data is a user profile and relates to an entity that is included as part of the repository network service provided by the repository server130. Further, according to some example embodiments, an entity typically relates to an individual programmer, but may relate to an organization, for instance a business or other group. In some example embodiments, a software developer profile includes at least a unique identifier (the identifier uniquely identifies the entity on the repository service), a name for the entity (typically a string of text, perhaps alphanumeric characters) and a plurality of links between the entity and other entities that form part of the repository webservice.

The links may be bidirectional in nature. For example, two software developers may collaborate on the same code project. Because the two developers work on the same coding project, they may be bidirectionally linked under the assumption that each knows of the other as a fellow coder (e.g., team member, colleague) on the project. The links may alternatively be unidirectional, e.g., the first software developer receives updates published by the second software developer but the second software developer does not receive updates published by the first software developer. In some embodiments, the data stored on the repository website indicates the type of communication activity between the users. For example, the node data may include an indication that a first user commented on a pending code update on a project page on the repository website. The links may indicate the another entity by including a identifier that is unique to the other entity. Typically, repository webservices provide an identifier that is an alphanumeric string. The string may be known to the entity and other users (e.g. it may be their username) or it may be a system-generated identifier which does not need to be known to the user (e.g. a string such as “exampleidentifier$43*”). The profile may also include a uniform resource locator (URL) that is unique to the entity.

A user profile of the repository website may also have other information associated with pre-defined fields, for instance ‘high school attended’, ‘place of residence’, ‘place of work’, ‘undergraduate study subject’, etc. The profile may also have other content such as photographs, videos, comments or profile text, etc. Profile content may be associated with particular dates (and as such may appear in a timeline on a user's profile page) or may not be dependent on a date (and so may not generally appear on a timeline). In some embodiments, profile content may be associated with geotagged data.

In some example embodiments, user profiles are imported in response to user input. For example, a first profile is imported by the network graph parser234in response to the user selecting a first entity in the repository service. This may occur for instance by the user selecting a hyperlink in a code projects webpage provided by the repository server130. The code projects webpage may be provided by the repository server130in response to the user entering text, e.g. the whole or part of a name on an entity, into a search field of a webpage provided by the by the repository server130. The code projects webpage displays coding projects of the first entity, where each of the coding projects has its own projects webpage, which can be spidered as described above. According to some example embodiments, upon selection of the first entity, the network graph parser234sends a request to the repository server130identifying the first entity. In response, the repository server130provides the code projects webpage of the first entity, which is then parsed by the network graph parser234. In some example embodiments, the network graph parser234extracts node data from the code projects webpage by accessing the source code (e.g., markup language) of the code projects webpage and then extracting the node data listed in the source code. The received node data is stored in volatile memory (e.g. RAM) allocated to the browser1632, but is not stored in permanent memory, e.g. ROM.

After the node data of the first entity is imported by the network graph parser234, or at least after importation has begun, the user selects a second entity. This may occur for instance by the user selecting a hyperlink relating to the second entity in a second code projects webpage provided by the repository server130. Upon selection of the second entity, the network graph parser234sends a request to the repository server130identifying the second entity. In response, the repository server130provides a second code projects webpage that lists all the coding projects for the second entity on the repository server. The parse engine220then parses the source code of the second code projects webpage to extract additional node data of the users associated with the second entity (e.g., users that have worked on the same coding project as the second entity). The received profile is stored in volatile memory (e.g. RAM) allocated to the browser1632, but is not stored in permanent memory, e.g. ROM.

According to some example embodiments, the network graph parser234is configured to automatically import node data for entities to which the first and second entities are linked, e.g., for which links from the first and second entities exist. The parse engine220is configured to import such node data by sending requests to the repository server130identifying the further entities and navigating to the code projects webpages of the entities.

At operation235, the node data engine230transforms the contributor data parsed from the projects webpage from a first format into a second format. For example, the underlying source code of the projects webpage may be a markup language, such as HTML. The node data parsed from the projects webpage may also be in the markup language format. The node data engine230is configured to transform the node data from the markup language format to an attribute-value format, such as JSON (JavaScript Object Notation). The node data in the second format can be used for filtering and generation of the visualizations.

At operation340, the visualization engine240creates a visual representation from the parsed node data (e.g., node data in the attribute-value format). In some example embodiments, the visual representation is generated as a network graph in an additional tab of the browser1632. The network graph includes a collection of nodes connected by edges. Each node corresponds to a user from one of the projects listed on a code projects webpage, and connections between individual nodes may be visually represented as lines, for example straight lines. In some example embodiments, two nodes are connected on the repository server if each of the nodes are associated with the same coding project. The graph may lend itself to be further processed, analyzed and manipulated by an analyst or other user. The details regarding operation340are explained in more detail later.

At operation350, the export engine250exports the graph formed from the operation340to the database system10. The database system10is connected to the electronic device110(as shown inFIG. 1), according to some example embodiments. Further, according to some example embodiments, the database system10is implemented as a backend system disposed in an air-gapped, high-side environment, separated from the network120and the electronic device110. The database system10may be dedicated to receive data for further analysis. Therefore, network graph parser234of operation340can be used to collect and pre-process the data such that it is compatible with the database system10.

FIG. 4is a flowchart representing a method400performed by the network graph parser234for importing node data of the first and second entities from the repository server130and for creating a graph, according to some example embodiments. The method400is an example of sub-operations performed to complete operations330and340ofFIG. 3discussed above.FIGS. 5A and 5Bshow an example of the graph created by the exemplary method400.

At operation410, the interface engine210receives selection of first entity through a user input, for instance through a bookmark, favorite, or through selection of an option provided in a list of search results. At operation420, the interface engine210requests the profile of the first entity. This involves the network graph parser234accessing the repository server130via the network120and in particular accessing the first entity (e.g., projects webpage of the first entity) in the repository server130. In particular, the network graph parser234may send an HTTP request to the repository server130, the request including the unique identifier of the first entity.

At operation430, the network graph parser234receives the profile or projects webpage of first entity. The profile is for example received as an HTTP response. According to some example embodiments, the profile includes a name for the first entity and details of connections of the first entity. The connections define links to other entities, and include unique identifiers for the other entities. In some example embodiments, one or more webpages of the first entity may be exposed through automatic scrolling of the one or more webpages. For example, a top portion of a first entity's webpage may be initially retrieved, and further portions below the top portions may be auto populated by script as those portions are scrolled to. In some example embodiments, the auto populated scrolled-to portions are received at operation430.

At operation440, the visualization engine240displays a graph relating to first entity. For example, the network graph parser234may display a group or ‘cloud’ of nodes, each node relating to an entity. The node relating to the first entity is displayed with different visible characteristics to nodes for other entities. For instance, it may be a different color or size. All the nodes for entities linked to the first entity are shown as being connected by the inclusion on the graph of a line, e.g. a straight line, connecting the node to the node for the first entity. In some embodiments, connections between nodes other than connections between the first entity and other nodes may not be displayed in the graph.

Further, in some example embodiments, a further entity (e.g., a second entity of operation460) need not be specified for links between nodes to be created. For example, an entity associated with a given code repository page may be identified (e.g., at operation410). The code repository page may list other coding projects with which the entity is involved (e.g., develops code). Each coding project may list other further entities associated with the given project. Using the identified entity, the additional projects and additional entities can all automatically be included in a single network graph, according to some example embodiments.

In the following discussion, the terms ‘connected’ and ‘linked’ in relation to entities included in the electronic repository website can be used interchangeably.FIG. 5Ashows an example of parsing profiles of users. In the left panel ofFIG. 5A, a node501is displayed as an empty circle. The node501corresponds to the first entity. Each of ten nodes displayed as a group around the node501represents a different entity to which the first entity is linked or connected, as identified from the profile of the first entity. Each such node in the group (other than the first node501) is connected to the node501with a respective straight line, which represents a link between the corresponding two entities. In the following, a group of nodes connected to the first node501once by a single link in the created graph may be represented as being enclosed within a dotted circle, as shown in the right panel ofFIG. 5A.

At operation450, the network graph parser234begins requesting profiles of entities linked to by the first entity. In some example embodiments, the profiles are parsed from a code projects webpage of the first entity. For example, users associated with the first entity may be displayed in a projects webpage. The underlying markup language of the code projects webpage can be parsed to extract the username, user profile URL, and other information for each of the users associated with the first entity.

At operation455, profiles of the entities are stored as they are received. In one embodiment, the profiles are stored in non-volatile memory that is allocated to the browser1632. Profiles may continue to be requested and saved as a background task whilst the network graph parser234performs other tasks.

At operation460, the interface engine210receives selection of a second entity. This may occur as described above in relation to receiving selection of the first entity.

At operation470, the network graph parser234receives the profile of the second entity, after requesting the profile of the second entity. The profile is for example received as part of an http response. The profile includes at least a name for the second entity and details of connections of the second entity. The connections define links to other entities, and include unique identifiers for the other entities.

At operation490, the visualization engine240displays a graph relating to the first and second entities. For example, the network graph parser234may display three groups (or clouds) of nodes510,520,530, each node relating to an entity. The nodes501and502relating to the first and second entities are displayed with different visible characteristics to nodes for other entities. For instance, they may be a different color or size. Each node of the first group530of nodes corresponds to an entity linked to in the profiles of both the first and second entities. Each node of the second group510of nodes corresponds to an entity linked to by the profile of the first entity but not by the profile of the second entity. Each node of the third group520of nodes corresponds to an entity linked to by the profile of the second entity but not by the profile of the first entity. All the nodes for entities connected to the first entity are shown as being connected to the node501by the inclusion on the graph of a line, e.g. a straight line, connecting the node to the node for the first entity. All the nodes for entities connected to the second entity are shown as being connected to the node502by the inclusion on the graph of a line, e.g. a straight line, connecting the node to the node for the second entity. Connections between nodes other than connections between one of the node501and the node502and other nodes are not displayed in the graph.

At operation490, the visualization engine240creates a new graph after removing the graph as shown inFIG. 5A. Alternatively, the network graph parser234may augment and rearrange the graph created at operation440. An example of the created graph at operation490is shown inFIG. 5B. Nodes501and502represent the first entity and the second entity, respectively. Third group510comprising seven nodes (represented as filled circles), correspond to the entities linked to by both the first entity and the second entity. The second group510comprise only three nodes because seven nodes previously in the sole group now belong to the first group530. The profile of the second entity includes links to fourteen entities. Seven entities linked to in the profile for the second entity belong to the first group530, and the other seven entities belong to the third group520. The nodes of the group fromFIG. 5Ais now split and rearranged into two groups, namely second group510and first group530. Therefore, in displaying nodes corresponding to the entities linked to by the first entity501and second entity502, accessed by the network graph parser234, they are grouped into three groups: the second group510linked only to the first entity501, the third group520linked only to the second entity502, and the first group530linked both to the first entity501and second entity502.

At operation495, the interface engine210begins requesting profiles of entities linked to by the second entity. In some example embodiments, the operations of450and495(e.g., requests for profiles of related entities) are initiated by a manual user request. For example, after the user (at operation420) requests profile of first entity, the user (at operation450) further requests (e.g., using a GUI button) the profiles of entities related to the first entity. Further, according to some example embodiments, the operations of450and495are performed automatically by the network graph parser. For example, after the user (at operation420) requests profile information of the first entity, the network graph parser234automatically retrieves and sends the profile information of the specified first entity but also retrieves and sends profile information of entities related to the first entity automatically (e.g., without the user manually initiating the request for profile information of the related entities).

At operation497, profiles of the entities are stored as they are received. In one embodiment, the profiles are stored in volatile memory, e.g. the RAM1606, that is allocated to the browser1632. Profiles may continue to be requested and saved as a background task whilst the network graph parser234performs other tasks. Further, according to some example embodiments, the display operations of method400(e.g., operations440and490) are bypassed until the some or all of the information collection operations (e.g., operations410,420,430,450,455,460,470,480,495, and497) are completed.

FIG. 6is a flowchart showing a method600performed by the network graph parser234for importing further seed entities from the electronic repository website system hosted from the repository server130and for creating a graph, according to some example embodiments. This may correspond to at least part of operations330and340ofFIG. 3, as discussed above.

FIG. 7shows an example of the graph created by the exemplary method600. Prior to operation610, the plugin is processing two or more entities, for instance as is shown inFIG. 5Band is present at the end of the flowchart ofFIG. 4. At operation610, the interface engine210receives selection of further entity through a user input, for instance through a bookmark, favorite or through selection of an option provided in a list of search results.

At operation620, the interface engine210requests or parses the profile of the further entity. This is similar to operation420. This involves the interface engine210accessing the repository server130via the network120and accessing first entity in one of the electronic repository webservice system in the repository server130. In particular, the interface engine210may send an HTTP request to the repository server130, the request including the unique identifier of the first entity. Alternatively, the network connection parser can parse a code projects webpage to extract profile information of user connected to the second entity.

At operation630, the interface engine210receives the profile of the further entity. This is similar to operation430. The profile is for example received as part of an HTTP response. The profile includes at least a name for the first entity and details of connections of the further entity. The connections define links to other entities, and include unique identifiers for the other entities.

At operation640, the visualization engine240displays a graph relating to all the selected entities. Here, the visualization engine240may cause display of multiple groups (or clouds) of nodes, each node relating to an entity. Each group relates to a collection of nodes that have the same connections to the selected entities. Where there are three selected entities, there are seven groups. Each node of the first group of nodes corresponds to an entity linked to in the profiles of both the first and second entities, but not the third entity. Each node of the second group of nodes corresponds to an entity linked to by the profile of the first entity but not by the profile of the second or third entities. Each node of the third group of nodes corresponds to an entity linked to by the profile of the second entity but not by the profile of the first or third entities. Each node of the fourth group of nodes corresponds to an entity linked to in the profiles of both the first and third entities, but not the second entity. Each node of the fifth group of nodes corresponds to an entity linked to by the profile of the second and third entities but not by the profile of the first entity. Each node of the sixth group of nodes corresponds to an entity linked to by the profiles of the second and third entities but not by the profile of the first entity. Each node of the seventh group corresponds to an entity linked to by each of the first, second and third entities. One or more of the groups may not exist, if there are no nodes that meet the criteria for that group (these groups might be said to have zero nodes).

The nodes relating to the selected entities are displayed with different visible characteristics to nodes for other entities. For instance, they may be a different color or size. All the nodes for entities connected to the one of the selected entities are shown as being connected by the inclusion on the graph of a line, e.g. a straight line, connecting the node to the node for the selected entity. Where a non-selected node has links to multiple selected entities, there is a line for each such connection. In some embodiments, connections between two nodes that relate to non-selected entities may be hidden or not displayed in the graph. In some example embodiments, the graph may simplify or de-clutter the graph by hiding links between nodes and/or nodes based upon whether a give node or one of its neighbors is selected. For example, if the user selects a given node, the visualization engine may only display notes that are directly linked to the given node.

At operation650, the interface engine210begins requesting profiles of entities linked to by the further entity. In some example embodiments, the user manually requests the profiles of entities linked to by the further entity. At operation660, profiles of the entities are stored as they are received. In one embodiment, the profiles may be stored in volatile memory that is allocated to the browser1632. Profiles may continue to be requested and saved as a background task whilst the network graph parser234performs other tasks. At operation670, the operation may check whether another entity has been selected by the user. If so, the operation returns to operation620, where the profile for the further selected entity is requested. Further, in some example embodiments, the selections of additional entities are processed in batches. For example, instead of requesting information of a single further entity and then receiving the information of the single further entity (e.g., method600), the user can select a plurality of entities, then request their information as a batch process (e.g., as part of a single request).

Further, according to some example embodiments, the display operation of method600may be bypassed or delayed until other operations are complete. For example, operation650(an information collection related operation) may be performed before operation640(a display related operation). As a further example, the information collected at operation650may be stored to memory and operation640is bypassed and a display is never generated).

FIG. 7shows a screenshot of an example of graph generated by the visualization engine240, according to some example embodiments. Here, six nodes701,702,703,704,705and706correspond to six entities that have been selected by a user. The nodes701,702,703,704,705and706corresponding to user-selected entities are displayed as empty circles. Entities linked to by the selected entities are represented as nodes, and are displayed as filled circles. A line connects each node pair representing linked entities if at least one of the linked entities is a user-selected entity. I

It can be seen fromFIG. 7that nodes are grouped together depending on which one combination of the six user-selected entities they are linked to. For example, the group of nodes711correspond to non user-selected entities linked to two of the user-selected entities701and705. Node712corresponds to the only non user-selected entity linked to by the user-selected entities702,703and706, hence the node712forms a group on its own. The group of nodes713includes nodes relating to entities linked to user-selected entities703and706. Each group is displayed separated from other groups, e.g. with a gap between the groups which is visibly significantly larger than the gaps between adjacent nodes forming part of a single group. Each group is displayed separately from one another to aid visual recognition of groups representing different states of connection. In some example embodiments, each node is generated from node data from the same code repository website. In some example embodiments, some of the nodes are generated from node data from a first code repository website and some of the nodes are generated from node data from a second code repository website different from the first. In this way, an analyst user can determine relationships between nodes (e.g., software project data) across different code repository websites.

FIG. 8is a flowchart representing an exemplary method800performed by the network graph parser234to generate a list in the form of a histogram from the imported profiles. The method800is performed when a graph relating to at least one selected entity is provided for display by the network graph parser234and when the profiles for all of the selected entities and the entities linked to the selected entities have been received from the repository server130. The histogram may be provided in response to a user input selecting a histogram option, for instance through interaction with a user interface element in a sidebar, dock, pull-down menu etc.

FIG. 9shows an example of a graph900generated by the method600. It also shows an example of a histogram990created by the exemplary method800. It further shows a profile viewer995generated by selecting a node displayed in the graph900. The graph900, the histogram990, and the profile viewer995are displayed at the same time on different parts of the display212, for instance in the layout shown in the Figure. The graph900has been generated from the imported profiles of three user-selected entities, corresponding to displayed nodes901,902and903. As explained inFIG. 5aand the accompanying paragraphs above, the circles910,920and930represents first, second, third group of nodes corresponding to entities directly linked only to the user-selected entity nodes901,902,903, respectively. There are three more groups of nodes904,906,908, which correspond to entities linked to only two of the selected entities901,902and903. There is one group905of nodes linked to all three of the selected entities901,902and903.

At operation810, the network graph parser234selects one of the fields of a profile relating to one of the selected entities901,902,903. In this example, the profile contains fields of information common to all or many of the profiles such as place of birth, birth year, high school, and place of work.

At operation820, the node data engine230then searches in all or selected imported profiles for profiles which have the same information in the same field. In particular, the node data engine230identifies which fields of the profile of the selected entity are populated. For a populated field, the plugin extracts the information (text, numbers or text and numbers) from the profile and searches the corresponding field of all the other profiles for the same information. Since the profiles for the entities are stored in the volatile memory allocated to the browser1632, this searching can be relatively fast.

At operation830, the node data engine230generates a record indicating any other entity which has the same information in the same field of the profile. The record is made in the working (volatile) memory206allocated to the web browser1632.

At operation840, the node data engine230determines whether there are other fields in the profile for the selected entity that include information and that have yet to be processed. If there are such other fields, then the method proceeds to operation850, where another field is selected, before the method returns to operation820. If all the fields have been processed, the method proceeds to operation860.

At operation860, the node data engine230determines whether all the selected entities have been processed. If not, then the next entity is selected for processing at operation870and the method then returns to operation810. If so, then at operation880the visualization engine240generates a histogram from the processed data. According to some embodiments, operation880is reached only when all completed fields for the selected entities (the entities which have been selected by a user in the method300, the method400or the method600).

According to some example embodiments, operation880involves identifying counting the number of profiles with the same information in the same field, and forming a list. The list may ordered according to the count of profiles or by a value of the field. Following operation880, the histogram is displayed on a display screen of electronic device110at operation890. Operations810to880may be performed by the network graph parser234without the user having requested a histogram, according to some example embodiments. In this case, however, the histogram may be displayed at operation890only in response to the option having been selected by the user. InFIG. 9, an example of such a histogram990is shown. In this example, the items in the profile description information shared by more than one entities in the graphs were A university, B high school, C high school, living in D city, living in E city, working at F company, working at G company and self-employment.

Returning toFIG. 9, at operation893, the interface engine210receives a user input selecting one of the items. In some embodiments, the user input may be in the form of the user clicking on the row of the histogram990. In some embodiments, the user input may be in the form of moving cursors to indicate the desired entry in the histogram. In the example shown inFIG. 9, the user input has been received for ‘Lives in E city,’ which is shared by five entities corresponding to nodes displayed in the graph. At operation896, in response to this user input, five nodes corresponding to the five entities sharing the profile description information ‘Lives in E city’ are highlighted. The five entities are treated as being participants in the “Lives in E city” group”; that is, the user's have the attribute of living in E city. InFIG. 9, the highlighted entities are represented by the differently colored nodes904,905,906,907and908.

At any time, any one of the nodes in the graph900may be selected by the user using the input device214and the cursor control216. Once selected, the profile995of the entity corresponding to the nodes may be displayed near the graph900. InFIG. 9, for example, when the entity corresponding to the node904, which is highlighted due to the fact that the profile indicates that the entity ‘lives in E city,’ is selected by the user, the profile view995may be generated and displayed near the graph900. The information included in the profile view995is present in the volatile memory allocated to the browser1632because the profile information was retrieved from the repository server130during performance of the method400, the method600or the method800.

FIG. 10is a flowchart showing a method1000performed by the network graph parser234of the electronic device110to provide a search facility which can be used to search the profiles of the imported entities, according to some example embodiments. The search facility may be provided in response to a user input selecting a search facility option, for instance through interaction with a user interface element in a sidebar, dock, pull-down menu etc. At operation1010, the network graph parser234may generate a search tool1150which can receive a user input for a keyword. In the example ofFIG. 11, the keyword ‘E city’ is input into a text entry box provided by the search tool1150. The keyword ‘E city’ corresponds to a group of users that live in the city called ‘E city”.

FIG. 11shows an example of a graph1100generated by the method600and an example of a search tool1150generated and operated by the exemplary method1000. In the left panel ofFIG. 11, an example of a graph1100generated by the method600is shown. In this example, the graph1100is similar to the graph900inFIG. 9. The graph1100is generated from the imported lists of three accessed entities1101,1102and1103. The search tool1150may provide any form of user interface element that can receive the input of the user from the input device214. For instance, the search tool1150may provide a text box into which a user can type alphanumeric characters such as a word or words.

At operation1020, the node data engine230may search in the profiles of the imported entities in the generated graph1100which have an entry that matches with the keyword input in the search tool1150. This is performed by searching the information in the profiles as stored in the working volatile memory allocated to the browser1632. At operation1030, if one or more profiles are found to have the same text as the input text, the method proceeds to operation1040. Here, the corresponding nodes in the graph1100are highlighted via the visualization engine240. If not, the result of search is reported at operation1050. In the example ofFIG. 11, five entities1104,1105,1106,1107and1108, are highlighted as a result of the search for the keyword ‘E City.’

FIG. 12is a flowchart showing a method1200performed by the network graph parser234of the electronic device110to filter the data associated with the entities in a plotted graph to produce a reduced graph, according to some example embodiments.FIG. 13shows examples of graphs1300generated by the method600and examples of reduced graphs1310(e.g., a refined visual representation) and1320generated by the exemplary method1200.

The filter instruction may be provided in response to a user input selecting a filter option, for instance through interaction with a user interface element in a sidebar, dock, pull-down menu etc. If the number of entities displayed in the graph1300is large, the graph may be of limited use to an analyst. The filtering method1200allows the isolation of the most significant entities and the removal of less significant entities. Such operation of filtering or reducing data may lead to more efficient, focused and targeted approach in repository website user analysis. This applies to analysis using the network graph parser234and to subsequent analysis after export to the database system10. Furthermore, trimming the graph before exporting data to the database system10may prevent the personal profile data of only marginally relevant or irrelevant individuals unnecessarily entering into the database system10for analysis. It may also provide regulation compliance advantages since information relating to fewer entities is imported into the database system10.

At operation1210, the interface engine210generates a user interface element1350configured to receive a user input specifying a connection parameter, such as a minimum number of links that is of interest to the user (e.g., a level of connectedness). Limiting the minimum number of links may assist in selecting the entities with the most meaningful connections in the network represented in the graph1300. The user interface element1350may receive the user input via the input device1614or the cursor control1616.

At operation1220, the node data engine230identifies the entities linked to other entities by the number of connections specified by the user input at operation1210. All of the connections inFIG. 13correspond to links to one of the selected entities1301,1302and1303. Therefore, the number of links of an entity in the example ofFIG. 13only corresponds to the number of connections to the user-selected entities1301,1302and1303.

In the example ofFIG. 13A, the maximum number of links between entities is three. Therefore, the user input may be “2” or both “2” and “3”. The user input of both “2” and “3”, as shown inFIG. 13A, may instruct the node data engine230to identify the entities with two and three links to selected entities. The user input of “3”, as shown inFIG. 13B, may cause the network graph parser234to identify only the entities with links to three selected entities. Returning toFIG. 12, at operation1230, the node data engine230searches the nodes (e.g., underlying node data in JSON format) corresponding to the identified entities. InFIG. 13A, as a result of search in this operation, the nodes corresponding to entities having two and three links with the user-selected entities1301,1302and1303, corresponding to groups of nodes1304,1305,1306and1307, have been highlighted by displaying them as empty circles.

InFIG. 13B, the entities having two links with the user-selected entities1301,1302and1303, corresponding to group of node1305, have been highlighted as empty circles. Returning toFIG. 12, at operation1240, the entities that are not identified at operation1230and that are not the user-selected entities1301,1302and1303may be removed from the graph1300, according to some example embodiments. This may be achieved by the network graph parser234receiving a user input to ‘inverse select’ the other entities that are not highlighted At operation1230, and then receiving an input to delete the selected nodes/entities, the delete input being received via the input device1614or the cursor control1616. Alternatively, the network graph parser234may receive a user input (e.g., a filter instruction) to remove all the entities except the highlighted entities at operation1230and the user-selected entities1301,1302and1303.

FIG. 13Ashows an example of a graph1310reduced from the graph1300according to the method1200. In the user interface element1350, “2” and “3” links have been specified by the user and the graph1310shows only the user-selected entities1301,1302and1303and the entities that are linked to two or three of the accessed entities, groups of nodes1304,1305,1306and1307. InFIG. 13B, the graph1320shows an example of a graph trimmed from the graph1300according to the exemplary method1200. In the user interface element1350, “3” links have been specified and the graph1320shows only the user-selected entities1301,1302and1303and the entities that are linked to all three of the user-selected entities, namely the group of nodes1305.

In case the profile description information have been imported along with the entities in the graph1300, they may be removed along with the entity at operation1240. After operation1240, the reduced graphs1310or1320and/or associated profile description information may be exported to the database system10via export engine250. Though visual graphs are depicted inFIGS. 13A and 13B, it is appreciated that the operations may first be performed on the underlying data used to generate the graphs. That is, the graph1300may be generated from initial node data collected from a connections page. A connection parameter may be received from the user that specifies the number of connections required to remain in the node data. Nodes not meeting the attribute specified by the connection parameter are removed. The resulting refined node dataset is then used to generate graph1310.

FIG. 14is a flowchart representing an exemplary method1400performed by an electronic device110to export the data associated with the entities in the reduced graphs1310and1320. This may correspond to operation350discussed above in relation toFIG. 3. At operation1410, the network graph parser234may receive a user input which instructs the network graph parser234to export the reduced graphs1310or1320and associated data such as profile description information of the entities corresponding to the nodes displayed in the graphs1310or1320.

At operation1420, the interface engine210receives a user input specifying an analysis description. The analysis description may be free text. It may relate to the origin, the history and the description of the data and the details regarding the repository website analysis performed. The analysis description may assist in generating trails such that it can be monitored that the performed analysis complies with any rules or regulations that may be relevant in the specific field of analysis. The analysis description also may be useful in case multiple sets of reduced and processed graphs are generated from different starting accessed entities, for example. If a specific entities appear in multiple sets of graphs, the analysis description of each graph may provide additional information therefore provide compounding value of multiple investigations.

At operation1430, the network graph parser234may export the data to the database system10via export engine250. Operation1430may involve exporting data relating to entities corresponding to nodes displayed in the graph to the database system10without exporting data relating to entities corresponding to nodes not displayed in the graph. In the database system10, the reduced graph and the associated data may be transformed according to the specific ontology of the deployment for further analysis.

Various modification and alternatives will be apparent to the person skilled in the art and all such modifications and alternatives are intended to be encompassed with the claims Some such modifications and alternatives will now be described.

Although in the above, the profiles for the user-selected entities are sourced from the same electronic repository website service provider, the scope is not limited to this. In other embodiments, profiles for an entity may be retrieved from two or more different repository servers130-1to130-n. In this case, the entity would ordinarily have different identities or usernames on the different electronic repository websites. However, the profiles can be determined by the network graph parser234to be related to the same entity by information included in either profile or in both profiles, or may be entered into the network graph parser234by the user of the network graph parser234. Alternatively or in addition, two or more different entities from different electronic repository servers130may be selected by the user of the network graph parser234as seed entities. In this case, information in profiles for linked to entities may be used to connect profiles in one or more of the repository servers (e.g., repository server130-1) to corresponding profiles for the same entities in another repository server (e.g., repository server130-2).

In the above, when an entity is selected for analysis, all of the entities linked to by that profile are retrieved from the electronic repository server130and displayed in a graph. Alternatively, a user may specify a limit on the number of entities that are to be retrieved from the electronic repository server130by the network graph parser234and displayed in a graph. This may be globally set as a setting by the plugin, or it may be selected or entered by the user at the time of selecting the entity. In the above, the histogram is formed from same information in same fields or profiles. Alternatively or in addition, information such as geotag information from photos, comments, mentions, replies, and/or such like.

FIG. 15Ashows an example browser1500for parsing node data using the network graph parser234, according to some example embodiments. In the example ofFIG. 15A, an analyst user navigates to the user profile of a user on a code repository website. For example, the analyst user navigates to the URL1505(“repository/joan.labrador/”), which is a projects webpage of the software developer “Joan Labrador”. In some example embodiments, the analyst user is a user attempting to identify patterns between software projects and the software developer is a user that uploads the source code to a project webpage of a given software project.

The projects webpage displays the user's uploaded software or project data1510as display elements (e.g., boxes, static text, hyperlinks). The title for each of the projects may contain a hyperlink that links to the project page for the corresponding project. For example, in the first listed project, “Smartwatch Exercise App” may be a hyperlink that links to a project page for that project. The project page for “Smartwatch Exercise App” may display source code uploaded by the software developer “Joan Labrador”. The project page may further contain links to the user profile pages of the seventeen developers that work on that project.

The projects webpage is received as HTTP data from the repository server130. The webpage is generated from underlying source code in a format, such as HTML. To initiate parsing, the analyst user selects a plugin button1515which, as displayed, is integrated into the browser1500. Responsive to the selection, the interface engine210displays a popup window1520having different parse options. According to some example embodiments, the first option “Graph” parses all users associated with the user “Joan Labrador” and creates a visualization from the data as discussed above. The second option “Add to graph” adds Joan Labrador as a second entity. For example, the analyst user may have selected a first user to parse (e.g., collect node data of related developers), and then want to select Joan Labrador as a further entity to parse (e.g., collect node data of developers related to Joan Labrador to add to the graph).

Assuming, to continue the example, the data analyst selects the first option “Graph”, the network graph parser234parses the source code that generates the projects webpage to extract node data from Joan's projects as discussed above. For example, the parse engine220can identify each of Joan's projects, including (1) “Smartwatch Exercise App”, (2) Java Note Taking client”, and (3) “Acme Corp. Enterprise CRM System”. The parse engine can navigate to the project page for each of the projects to identify users associated with Joan. For example, the parse engine220can user the hyperlink “Smartwatch Exercise App” to navigate to the project page for that project. Further, the parse engine can then identify user profile links on the project page (e.g., the17developers working on the “Smartwatch Exercise App” project) and navigate to the user pages to collect node data such as user name, profile page URL, for each of the associated users. The parse engine may perform similar operations to collect node data for the users associated with the other two code projects. The resulting data can then be used to generate visualizations, as shown inFIG. 15B.

InFIG. 15B, displays a user interface1550showing a visualization1555generated from the node data of users associated with Joan Labrador through one or more coding or software projects. Each circle or node corresponds to a user associated with Joan through a project. The user interface1550may open in a second tab of the browser1550. As illustrated, the user interface1550includes a main area in which the visualization is displayed, and a right bar area1570. For example, selecting one of the buttons may display the user interface element1350(FIG. 13B) which the analyst user can use to specify a connection parameter. Father, as illustrated in the example ofFIG. 15B, the right bar area1570can be used to show parsed node data1557of the selected entity “Joan Labrador.” The parsed node data1557may be parsed or extracted from the underlying source code of the webpage displayed inFIG. 15A(e.g. a user profile page). According to some example embodiments, if a user select a node from the visualization1555, the corresponding node data for the node is shown in the right bar area1570.

Further, according to some example embodiments, the right bar area may be used to show other types of visualizations, such as the histogram990, instead of the node data. The analyst can then user the histogram to select groups to modify the visualization1555. In some example embodiments, the network graph parser spiders to one or more hyperlink for each users listed in a project page and to collect parsed node data similar to Joan's parsed node data1557.

FIG. 16is a block diagram that illustrates a computer system1600, which may constitute the electronic device110, according to some example embodiments. As illustrated, computer system1600includes a bus1602or other communication mechanism for communicating information, and one or more hardware processors1604(including processor circuitry), coupled with bus1602for processing information. One or more hardware processors1604can be, for example, one or more general purpose microprocessors, each including processor circuitry. Computer system1600also includes a main memory1606, such as a random access memory (RAM) or other dynamic storage device, coupled to bus1602for storing information and instructions to be executed by processor1604.

Main memory1606also can be used for storing temporary variables or other intermediate information during execution of instructions to be executed by processor1604. Such instructions, when stored in non-transitory storage media accessible to one or more processors1604, render computer system1600into a special-purpose machine that is customized to perform the operations specified in the instructions. Main memory1606may also be used for temporarily storing the whole of part of applications, such as the web browser1632, including the network graph parser234, while they are being executed by the electronic device110. As illustrated inFIG. 2, the network graph parser234may be integrated or installed into the web browser1632. For example, the network graph parser234may be installed as a plugin or extension of the web browser1632.

The main memory1606is a volatile memory in that data stored therein is lost when power is no longer provided to the memory1606. The main memory1606is used to temporarily store information that is being processed by software applications, including the web browser1632and the network graph parser234. In relation to the web browser1632and the network graph parser234, information that is temporarily stored includes webpages and ancillary content that is received from the repository servers130-1to130-n. In relation to the web browser1632and the network graph parser234, information that is temporarily stored also includes information parsed from webpages by the network graph parser234and information derived from such received information by the plugin, as is described in detail below.

Computer system1600further includes a read only memory (ROM)1608or other static storage device coupled to bus1602for storing static information and instructions for processor1604. The ROM1608is used for permanent storage of applications such as the web browser1632, including the network graph parser234, when the electronic device is not powered on and/or when the applications are not being executed by the processor1604. The storage is of the computer code or instructions that constitute the applications. A storage device1610, such as a magnetic disk, optical disk, or USB thumb drive (Flash drive), etc., is provided and coupled to bus1602for storing information and instructions.

Computer system1600can be coupled via bus1602to a display1612, such as an LCD or plasma display, or a touchscreen or cathode ray tube (CRT), for displaying information to a computer user. An input device1614, for instance a keyboard, including alphanumeric and other keys, is coupled to bus1602for communicating information and command selections to processor1604. Another type of user input device is cursor control1616, such as a mouse, a trackball, or cursor direction keys for communicating direction information and command selections to processor1604and for controlling cursor movement on display1612. In some embodiments, the same direction information and command selections as cursor control may be implemented via receiving touches on a touch screen without a cursor. It will be appreciated that the processor1604, under control of software and/or operating system, causes display of graphics and text, and that the display1612displays such. Displaying a graph comprises displaying a graphical representation.

Non-transitory media is distinct from, but can be used in conjunction with, transmission media. Transmission media participates in transferring information between storage media. For example, transmission media includes coaxial cables, copper wire and fiber optics, including the wires that comprise bus1602. Transmission media can also take the form of acoustic or light waves, such as those generated during radio-wave and infra-red data communications. Various forms of media can be involved in carrying one or more sequences of one or more instructions to processor1604for execution. For example, the instructions can initially be carried on a magnetic disk or solid state drive of a remote computer. The remote computer can load the instructions into its dynamic memory and send the instructions over a telephone line using a modem. A modem local to computer system1600can receive the data on the telephone line and use an infra-red transmitter to convert the data to an infra-red signal. An infra-red detector can receive the data carried in the infra-red signal and appropriate circuitry can place the data on bus1602. Bus1602carries the data to main memory206, from which processor1604retrieves and executes the instructions. The instructions received by main memory1606can optionally be stored on storage device1610either before or after execution by processor1604.

Computer system1600also includes a communication interface1618coupled to bus1602. Communication interface1618provides a two-way data communication coupling to a network link1621that is connected to a local network1622. For example, communication interface1618can be an integrated services digital network (ISDN) card, cable modem, satellite modem, or a modem to provide a data communication connection to a corresponding type of telephone line. As another example, communication interface1618can be a local area network (LAN) card to provide a data communication connection to a compatible LAN. Wireless links can also be implemented. In any such implementation, communication interface1618sends and receives electrical, electromagnetic or optical signals that carry digital data streams representing various types of information.

Network link1621typically provides data communication through one or more networks to other data devices. For example, network link1621can provide a connection through local network1622to data equipment operated by an Internet Service Provider (ISP)1626. ISP1626in turn provides data communication services through the world wide packet data communication network now commonly referred to as the “Internet”1628. Local network1622and Internet1628both use electrical, electromagnetic or optical signals that carry digital data streams. The signals through the various networks and the signals on network link1621and through communication interface1618, which carry the digital data to and from computer system1600, are example forms of transmission media.

Computer system1600can send messages and receive data, including program code, through the network(s), network link1621and communication interface1618. In the Internet example, a server1627might transmit a requested code for an application program through Internet1628, ISP1626, local network1622and communication interface1618. The received code can be executed by processor1604as it is received, and/or stored in storage device1610, or other non-volatile storage for later execution.

The network graph parser234is integrated into the web browser1632to form part of the web browser1632. The user can first download the network graph parser234from an appropriate web site or other source (e.g. portable storage such as a thumb drive or a storage device on a local network) and then can proceed to install the network graph parser234. Since a typical network graph parser234is designed to be compatible to a specific web browser1632(e.g., Google™ Chrome™, Mozilla™ Firefox™, Microsoft™ Internet Explorer™, etc.), the network graph parser234can become a part of the web browser1632automatically after the network graph parser234is installed.

Above, various actions are described as being performed by the network graph parser234and/or the web browser1632. It will be appreciated that this is shorthand for computer program instructions that form part of the network graph parser234or the browser1632, as the case may be, being executed by the processor1604and causing the processor1604to take the action. In doing so, some or all of the computer code/instructions constituting the network graph parser1634and the browser1632are copied from the ROM1608and stored in the main memory206, which is a volatile memory, such that the computer code/instructions constituting the network graph parser234and the browser1632can be executed by the processor1604. In executing the computer code/instructions constituting the network graph parser234and the browser1632, the processor204is controlled to store data (other than the computer code/instructions constituting the network graph parser234and the browser1632) temporarily in the main memory1606. As mentioned above, the main memory1606is volatile memory and as such data stored therein is lost when the main memory1606is de-powered.

Certain embodiments are described herein as including logic or a number of components, modules, or engines. Engines can constitute either software engines (e.g., code embodied on a machine-readable medium) or hardware engines. A “hardware module” is a tangible unit capable of performing certain operations and can be configured or arranged in a certain physical manner. In various example embodiments, one or more computer systems (e.g., a standalone computer system, a client computer system, or a server computer system) or one or more hardware engines of a computer system (e.g., a processor or a group of processors) can be configured by software (e.g., an application or application portion) as a hardware module that operates to perform certain operations as described herein.

The performance of certain of the operations may be distributed among the processors, not only residing within a single machine, but deployed across a number of machines. In some example embodiments, the processors or processor-implemented modules can be located in a single geographic location (e.g., within a home environment, an office environment, or a server farm). In other example embodiments, the processors or processor-implemented modules are distributed across a number of geographic locations.

The modules, methods, applications and so forth described in conjunction withFIGS. 1-15are implemented in some embodiments in the context of a machine and an associated software architecture. The sections below describe representative software architecture and machine (e.g., hardware) architecture that are suitable for use with the disclosed embodiments.