Source: https://patents.google.com/patent/US9003023B2/en
Timestamp: 2019-09-17 15:20:17
Document Index: 213817416

Matched Legal Cases: ['art 1302', 'art 1304', 'art 1302', 'art 1302', 'art 1402', 'art 1402', 'art 1402', 'art 1302', 'art 1402', 'art 1702', 'art 1702', 'art 2302', 'art 2404', 'art 2404', 'art 2302', 'art 2302', 'art 2302', 'art 2404', 'art 2602', 'art 2704']

US9003023B2 - Systems and methods for interactive analytics of internet traffic - Google Patents
Systems and methods for interactive analytics of internet traffic Download PDF
US9003023B2
US9003023B2 US13/495,537 US201213495537A US9003023B2 US 9003023 B2 US9003023 B2 US 9003023B2 US 201213495537 A US201213495537 A US 201213495537A US 9003023 B2 US9003023 B2 US 9003023B2
US13/495,537
US20130339514A1 (en
Upasona Kath HAZARIKA
2012-06-13 Application filed by Zscaler Inc filed Critical Zscaler Inc
2012-06-13 Priority to US13/495,537 priority Critical patent/US9003023B2/en
2012-06-13 Assigned to ZSCALER, INC. reassignment ZSCALER, INC. ASSIGNMENT OF ASSIGNORS INTEREST (SEE DOCUMENT FOR DETAILS). Assignors: CRANK, SAMUEL JOHN, HAZARIKA, UPASONA KATH
2013-12-19 Publication of US20130339514A1 publication Critical patent/US20130339514A1/en
2015-04-07 Publication of US9003023B2 publication Critical patent/US9003023B2/en
A computer implemented method, a cloud system, and a log system provide interactive analytics providing various intuitive mechanisms for interaction with data visualizations of Internet traffic, email, etc. The methods and systems utilize a cloud based monitoring system where all traffic from an organization may be monitored in a location and platform independent manner. The methods and systems include context-aware drilldown with progressively applied filtering and grouping while maintaining workflow history such that a user can go back to any point in the flow and proceed down a new path of investigation.
Generally, the field of art of the present disclosure pertains to computer and network systems and methods, and more particularly, to systems and methods for interactive analytics of Internet traffic such as web browsing statistics, email, and the like for providing various intuitive mechanisms for interaction with data visualizations.
Enterprises with computer networks have a need to monitor network usage for a variety of reasons including policy compliance, reporting, threat detection, determining efficiencies, and the like. Various conventional systems and methods exist for computer security, analytics, monitoring, etc. Varying degrees of analytics functionality exist in conventional systems and methods but these do not offer a comprehensive workflow that ranges from investigating a trend to taking an action for remediation. For example, one conventional system and method enable administrators to drill down from a high level report based on a grouping and then pivot on the given data. An administrator may view data grouped by Risk Class and then have the ability to view data grouped by domains for a given Risk Class. While this drilldown capability provides interesting data, there is no workflow and the administrator's drilldown activity is not tracked. As a result, the administrator is not able to report back on the various steps of the investigation. With the emergence of the cloud and use of the cloud for security, monitoring, etc., there is a need to provide functionality of systems and methods for interactive analysis of Internet traffic with cloud-based services.
In an exemplary embodiment, a computer implemented interactive analytics method includes logging data traffic between a plurality of users and an external network in a log system, wherein the logging is performed by a cloud based system; receiving a first request for interactive analytics of the logged data traffic, wherein the first request includes a date range, a visualization type, and a grouping; obtaining data traffic from the log system responsive to the first request; formatting the obtained data traffic based on the date range, the visualization type, and the grouping to provide a first interaction based on the first request; displaying the first interaction; and storing the first interaction in a history list.
In another exemplary embodiment, a cloud system for interactive analytics includes a plurality of servers forming a cloud system communicatively coupled to an external network and a plurality of users; and a log server communicatively coupled to the plurality of servers; wherein each of the plurality of servers include computer executable instructions, and, in response to execution, the computer executable instructions cause each of the plurality of servers to: monitor data traffic between each of the plurality of users and the external network; analyze the monitored data traffic; and log the analyzed data traffic to the log server in a compressed format; wherein the log server includes computer executable instructions, and, in response to execution, the computer executable instructions cause the log server to: present an interactive analytics tool to a user; receive requests via the interactive analytics tool for visualization of data in the log server; and maintain a workflow history in the interactive analytics tool for context-aware drilldown and progressively applying filters.
In yet another exemplary embodiment, a log system for interactive analytics includes a network interface communicatively coupled to a cloud based system, wherein the cloud system is configured to monitor each of a plurality of users independent of location, device type, and operating system of each of the plurality of users; a processor; and memory storing computer executable instructions, and in response to execution by the processor, the computer executable instructions cause the processing node to: receive monitored data from the cloud based system, wherein the monitored data includes all data traffic between a plurality of users and an external network; compress the monitored data; store the monitored data in a log file; receive a request for interactive analytics; obtain data from the log file based on the request; and maintain a workflow history of all requests for interactive analytics.
FIG. 6 is a flowchart of a network monitoring method for interactive analytics of Internet traffic;
FIG. 7 is a network diagram of an interactive analytics system for Internet traffic;
FIG. 8 is a flowchart of an interactive analytics method for interactive analytics of Internet traffic such as web browsing statistics and the like for providing various intuitive mechanisms for interaction with data visualizations;
FIG. 9 is a flowchart of an interactive analytics history method that can be used in conjunction with the method of FIG. 8;
FIGS. 10-19 are screen shots presented to a user of an exemplary operation of the interactive analytics systems and methods;
FIGS. 20-28 are screen shots presented to a user of another exemplary operation of the interactive analytics systems and methods; and
FIG. 29 is a flowchart of an interactive email analytics method.
In various exemplary embodiments, the present disclosure relates to systems and methods for interactive analytics of Internet traffic such as web browsing statistics and the like for providing various intuitive mechanisms for interaction with data visualizations. The interactive analytics systems and methods can be implemented through a cloud-based system. The interactive analytics systems and methods can include an analytics tool that provides a visual interface for interactive analysis of Internet traffic. Advantageously, the analytics tool enables visual exploration of data by providing various intuitive mechanisms for interaction with data visualizations. The Analytics tool supports interaction with data through various mechanism such as direct selection of visualization type, filters and grouping; context-aware drilldown, progressively applying filters and changing the grouping; history of steps taken, making it possible to go back to any point in the flow and proceed down a new path of investigation; and the like. The Analytics tool also provides several contexts in which to review the data: view the visualizations in real time during analysis, play back the analysis session multiple times, export the session to a file for offline review, and the like.
Referring to FIG. 1, in an exemplary embodiment, a block diagram illustrates a distributed security system 100. The system 100 can, for example, be implemented as an overlay network in a wide area network (WAN), such as the Internet, a local area network (LAN), or the like. The system 100 includes content processing nodes, processing nodes 110, that proactively detect and preclude the distribution of security threats, e.g., malware, spyware, viruses, trojans, botnets, email spam, data leakage, policy violations, etc., and other undesirable content sent from or requested by an external system. The system 100 can further monitor Internet traffic for all users connected thereto. Example external systems can include an enterprise 200, a computer device 220, and a mobile device 230, or any other network and computing systems communicatively coupled to the system 100. In an exemplary embodiment, each of the processing nodes 110 cam include a decision system and method, e.g., data inspection engines that operate on a content item, e.g., a web page, a file, an email message, or some other data or data communication that is sent from or requested by one of the external systems. In an exemplary embodiment, all data destined for or received from the Internet is processed through one of the processing nodes 110. In another exemplary embodiment, specific data specified by each external system, e.g., only email, only executable files, etc., is process through one of the processing node 110.
The system 100 can be configured to perform numerous functions including data aggregation, correlation, alerting, dashboard, compliance, retention, and the like. For example, the system 100 can aggregate data from many sources, including network, security, servers, databases, applications, providing the ability to consolidate monitored data to help avoid missing crucial events. The system 100 can perform correlation by looking for common attributes to link events together into meaningful bundles. Accordingly, the system 100 provides the ability to perform a variety of correlation techniques to integrate different sources, in order to turn data into useful information. The system 100 can use the automated analysis of correlated events for the production of alerts, to notify of immediate issues. The system 100 can provide dashboards tools which take event data and turn it into informational charts to assist in seeing patterns, or identifying activity that is not forming a standard pattern. The system 100 can be employed to automate the gathering of compliance data, producing reports that adapt to existing security, governance and auditing processes. Also, the system 100 can employ long-term storage of historical data to facilitate correlation of data over time, and to provide the retention necessary for compliance requirements.
The logging nodes 140 can include the logging node manager 148 that is communicatively coupled to data stores 142. As described herein, each of the logging nodes 140 can store data related to security operations and network traffic processed by the processing nodes 110 for each external system. In an exemplary embodiment, the logging nodes 140 can provide web logs of all visited URLs by user and the like. The logging nodes 140 can also log other data based on any number of parameters for future reporting, correlating, analysis, etc. The interactive analytics systems and methods provide a mechanism for visualizing data from the logging nodes 140 (or any other devices associated with the system 100).
The network interface 306 can be used to enable the server 300 to communicate on a network, such as the Internet, the WAN 101, the enterprise 200, and the like, etc. The network interface 306 can include, for example, an Ethernet card or adapter (e.g., 10BaseT, Fast Ethernet, Gigabit Ethernet, 10GbE) or a wireless local area network (WLAN) card or adapter (e.g., 802.11a/b/g/n). The network interface 306 can include address, control, and/or data connections to enable appropriate communications on the network. A data store 308 can be used to store data. The data store 308 can include any of volatile memory elements (e.g., random access memory (RAM, such as DRAM, SRAM, SDRAM, and the like)), nonvolatile memory elements (e.g., ROM, hard drive, tape, CDROM, and the like), and combinations thereof. Moreover, the data store 308 can incorporate electronic, magnetic, optical, and/or other types of storage media. In one example, the data store 308 can be located internal to the server 300 such as, for example, an internal hard drive connected to the local interface 312 in the server 300. Additionally in another embodiment, the data store 308 can be located external to the server 300 such as, for example, an external hard drive connected to the I/O interfaces 304 (e.g., SCSI or USB connection). In a further embodiment, the data store 308 can be connected to the server 300 through a network, such as, for example, a network attached file server.
The memory 422 can include any of volatile memory elements (e.g., random access memory (RAM, such as DRAM, SRAM, SDRAM, etc.)), nonvolatile memory elements (e.g., ROM, hard drive, etc.), and combinations thereof. Moreover, the memory 422 may incorporate electronic, magnetic, optical, and/or other types of storage media. Note that the memory 422 can have a distributed architecture, where various components are situated remotely from one another, but can be accessed by the processor 412. The software in memory 422 can include one or more software programs, each of which includes an ordered listing of executable instructions for implementing logical functions. In the example of FIG. 4, the software in the memory system 422 includes a suitable operating system (O/S) 426 and programs 428. The operating system 426 essentially controls the execution of other computer programs, and provides scheduling, input-output control, file and data management, memory management, and communication control and related services. The programs 428 can include various applications, add-ons, etc. configured to provide end user functionality with the mobile device 400. For example, exemplary programs 428 can include, but not limited to, a web browser, social networking applications, streaming media applications, games, mapping and location applications, electronic mail applications, financial applications, and the like..
Architecturally, the systems 100, 500 generally at least include the processing nodes 110 and the cloud nodes 502 for interfacing with the users 510, 520, 530, 220, 230. In an exemplary embodiment, all functions described herein can be incorporated in the processing nodes 110 and the cloud nodes 502. In another exemplary embodiment, the functions can be distributed. For example, the system 500 includes the singular cloud nodes 502 while the system 100 includes the processing nodes 110, the authority nodes 120, the logging nodes 140, etc. The term node as used herein can be a single computer, cluster of servers, or a functional module running on a computer, server, etc. Note, the cloud system 500 can perform data logging functionality at the cloud nodes 502.
Referring to FIG. 6, in an exemplary embodiment, a flowchart illustrates a network monitoring method 600 for interactive analytics of Internet traffic. The method 600 can operate over a sample interval (step 602). Note, the method 600 can also operate continuously with the sample interval just representing a break in collection. For example, the sample interval can be a minute, 15 minutes, etc. The method 600 monitors and collects data corresponding to system activity (step 604). As described herein in the context of the systems 100, 500, users are configured to communicate through the nodes 110, 502. In the method 600, the nodes 110, 502 are configured to collect data corresponding to system activity. The system activity can be any Internet request (e.g., URL request) or the like. The collected data corresponding to system activity can be encoded/compressed (step 606). Note, the system activity can include data related to an Internet request (e.g., User name, MAC address, IP address, URL, etc.), data related to a security event (e.g., blocked content, malware, virus, data leakage, etc.), data related to suspicious activity (e.g., legal liability, productivity loss, social networking, etc.), data related to file transfers (e.g., email, file transfer protocol, dropbox, etc.), data related to online chats or instant messaging (e.g., transcripts, etc.), data related to keyword monitoring (e.g., data leakage prevention, etc.), and the like. The encoded/compressed data is stored in a log (step 608). Finally, the method either starts a new sample interval (step 610) or continues in the same interval (step 610).
Examples of the compressed format are disclosed in commonly assigned U.S. patent application Ser. No. 12/185,443 filed Aug. 4, 2008 and entitled “ENCODING AND COMPRESSION OF STATISTICAL DATA,” the contents of which are incorporated by reference herein. As can be expected, the nodes 110, 502 can generate significant amounts of data based on multiple users being monitored continuously for all Internet activity. The purpose of the encoding/compression is to reduce log storage requirements. The compressed format can include Presence Vector, Differential Compression, Zero Byte Compression, Opcode compression, LZ77, Delta Compression, 1 bit compression, and the like. In operation, the data can include different data types such as constant data, time varying and predictable data, occasionally occurring data, time varying and linear data, whole records of data, time varying data, data that varies by one, and the like. For example, the following table provides examples of the types of compression that can be used for the different data types in the data.
Each record of the encoded/compressed data can have different fields corresponding to one of the data types listed above. For example, a user's identification (name, IP address, MAC address, etc.) would be constant data, i.e. the same for each field associated with the user. Of note, in studies, more than 80% of traffic leaving an enterprise is Hypertext Transfer Protocol (HTTP) or HTTPS (HTTP secure) traffic, and monitoring of this traffic generates massive amounts of logs. For example, a typical Fortune 500 company could generate 30-100 Gb of web logs every day. Further, companies typically have multiple Internet gateways generating multiple, disjointed web logs which fail to provide a consolidated view of overall corporate Internet activity. Companies retain such logs for a variety of reasons including to analyze security issues, to minimize IT downtime, to prevent system misuse, for compliance, etc.
The method 600 can, for example, be implemented in a processing node in the system 100 of FIGS. 1 and 2 or in a cloud node 502 of the system 500 of FIG. 5. In an exemplary embodiment, one advantage of the systems 100, 500 in implementing the method 600 is that all traffic from an organization's users can be monitored despite geographic differences (i.e., different offices in different geographic locations) and device differences (i.e., individual users having a desktop in the office, a laptop for travel/home use, a smart phone/tablet for mobile use, etc.). That is, an advantage of the cloud based systems 100, 500 is they can monitor the organization in an all inclusive manner regardless of location or platform. This provides the method 600 with an all encompassing view of an organization's traffic.
Referring to FIG. 7, in an exemplary embodiment, a network diagram illustrates an interactive analytics system 700 for Internet traffic. The system 700 includes one or more servers 702 coupled to logs 704. The servers 702 include an interactive analytics tool 706 operating thereon. The tool 706 can include software that enables a user 710 to perform interactive analytics on Internet traffic data stored in the logs 704. The user 710 can access the servers 704 through a network 712 which can include any of the Internet, a virtual private network (VPN), a local area network (LAN), a wireless LAN, or combinations thereof. The logs 704 can include data such as collected using the systems 100, 500 and the method 600. In an exemplary embodiment, the logs 704 can be Nanologs(tm) from Zscaler, Inc. (www.zscaler.com). The logs 704 can include all Internet traffic (HTTP/HTTPS) from all corporate users. In an exemplary embodiment, the user 710 can access the analytics tool 706 over an HTTP/HTTPS connection with the servers 702. That is, the analytics tool 706 can be via a web graphical user interface (GUI) to the user 710.
The analytics tool 706 is configured with state-of-the-art drill down capabilities, a complete end-to-end workflow while maintaining a historical trail of charts produced during the investigation. This enables the user 710 to go back at any point in time to review the details of each step of the investigation. Moreover, the user 710 may modify any step of the investigation and alter the path of the investigation from that point going forward—without having to restart the investigation from scratch. In addition, the analytics tool 706 enables the user 710 to generate am export file (such as in Portable Document Format (PDF)) which can then be sent to the relevant stakeholders that had requested the investigation—this may the HR department, a manager investigating one of his employees, or even a CIO looking for details on a developing traffic trend in the organization. Lastly, this analytics capability is available as a standalone feature but also can be initiated from any chart in the reporting interface by clicking on a segment of the chart and selecting “Further Analysis” which makes the tool even more versatile and powerful.
Referring to FIG. 8, in an exemplary embodiment, a flowchart illustrates an interactive analytics method 800 for interactive analytics of Internet traffic such as web browsing statistics and the like for providing various intuitive mechanisms for interaction with data visualizations. The method 800 can be implemented with the systems 100, 500, 700 and with the method 600. For example, the system 700 can be a subset of the systems 100, 500, and the systems 100, 500 can use the method 600 to collect data. The method 800 provides visualization of the data collected by the method 600 and stored in the logs 704 of the system 700. That is, the method 800 can be used for forensic and/or analytic investigations to mine enormous amounts of data in the logs 704. The method 800 allows organizations, through administrators, to analyze information about Internet use, such as employee activity, web mail and attachments sent, information published on social networking sites, instant messaging transcripts, etc. Thus, the method 800 allows organizations to mine the logs 704 for investigations, such as for regulatory reasons or internal inquiries. Through the method 800, administrators can see a drill-down of activities of specific periods of time, employees, departments, locations, and more. The steps shown in FIG. 8 for the method 800 can be executed independently and concurrently as appropriate and in any order.
The method 800 provides a visual interaction of data for a user. The method 800 includes selecting a date range (step 802). The date range can include the current day, current week, current month, previous day, previous week, previous month, or any custom range at any granularity (days, hours, minutes, seconds etc.). The method 800 includes selecting a visualization type for the interaction (step 804). For example, the visualization type is indicative of how the interaction is presented and can include a bar chart, a pie chart, a line chart, a totals lists, a summary table, a table of transactions, etc. In an exemplary embodiment, there can be a default visualization type such as a trend chart, and the user could start by adding a filter to the default type. Also, each time a new visualization is selected; a new chart can be created and included in a history list. The method 800 includes selecting data units for the interaction (step 806). The data units can include transactions (i.e., a number of transactions), data bytes, and in some cases time such as when a grouping is set to a domain. The method 800 includes selecting a grouping for the interaction (step 808). The grouping is indicative of the type of information for the interaction. For example, the grouping can include Application Class, Threat Class, Basic Threat Class, Basic Threat Type, Data Leakage Prevention (DLP) Class, DLP Dictionary, DLP Engine, Instant Message (IM) Direction, Location, Social Direction, Department, User Direction, URL Class, URL Category, URL Super-Category, Action, Web Application, Webmail Direction, Domain, etc.
With the date range, the visualization type, the data units, and the grouping selected, the method 800 creates the interaction, displays the interaction, and stores the interaction in a history list (step 810). Specifically, based on the selections, the method 800 obtains the requisite data from the logs 704, i.e. based on the date, data units, and grouping, and displays the interaction based on the visualization type. The method 800 also stores the interaction in a history list that stores a record of the order of the user's investigation. Once created, the user can change any of the aforementioned selections (step 812). If there is a change, the method 800 updates the interaction based on the change and stores the updated interaction in the history list (step 814). Once the interaction is finalized and presented to the user, the user can modify the presentation such as adding/removing a filter in the interaction (step 816). The filter can add/remove data in the interaction such as by Application Class, Threat Class, Basic Threat Class, Basic Threat Type, DLP Class, DLP Dictionary, DLP Engine, IM Direction, Location, Social Direction, Department, User Direction, URL Class, URL Category, URL Super-Category, Action, Web Application, Webmail Direction, etc.
For the transaction table visualization, the filters could include, for example, Action, Advanced Threat Type, Application Class, Basic Threat Class, Basic Threat Type, DLP Class, DLP Dictionary, DLP Engine, Direction, IM Direction, Social Direction, Streaming Direction, Threat Class, URL Category, URL class, URL Super Category, User, Web Application, Web Mail Direction, File Type, File Subtype, File Search, Received Byes, Sent Bytes, Total Bytes, Protocols, Location, Department, Threat Search, URL Search, Host Search, Path Search, User Agent, Referrer Search, Client IP, Server IP, etc. Additionally, the method 800 can include adding a filter and optionally setting a grouping based on a selection in the interaction (step 816). For example, the user could select one piece of visual data in the interaction and select a grouping for that visual data. Assume the visualization includes a graph of transactions grouped by URL class (e.g., general browsing, productivity loss, legal liability, etc.), application class (e.g., social networking, streaming content, etc.), etc. The user could select a particular graph and further drill down and group this content in a new interaction.
Referring to FIG. 9, in an exemplary embodiment, a flowchart illustrates an interactive analytics history method 900 that can be used in conjunction with the method 800 for interactive analytics of Internet traffic such as web browsing statistics and the like for providing various intuitive mechanisms for interaction with data visualizations. The method 900 can be implemented with the systems 100, 500, 700 and with the methods 600, 800. As described in the method 800, there is a history list of preceding interactions. The method 900 includes selecting one of a plurality of interactions in the history list (step 902). The selected interaction is displayed (step 904). That is, the user can select any previous interaction in the history list at any time. If the selected interaction is the last one in the history list (step 906), then any updates made to the selected interaction only affect it. That is, the method 900 can include updating the selected interaction based on a change and storing it again in the history list (step 908). If the selected interaction is not the last one in the history list (step 906), then any updates made to the selected interaction affect it as well as subsequent interactions in the history list. The method 900 can include updating the selected interaction based on a change and storing it again in the history list (step 910). Additionally, the method 900 can include deleting all subsequent interactions (from the selected interaction) in the history list based on the change (step 912). Thus, once a change or update is made in the middle or the beginning of the history list, all subsequent interactions can be automatically deleted.
The methods 800, 900 can be used to develop various reports for an organization such as a management report, an audit report, and a forensics report. As described herein, the organization can be a company, university, or any group. The organization includes a plurality of users which can each include a plurality of devices that access the Internet used a plurality of different mechanisms. Using the systems 100, 500 and the methods 600, 800, 900, the organization can monitor all Internet traffic of its users despite geographic differences, device and operating system differences, and the like. The management report can include high level charts that can be exported to PDF by an admin sends it to management on monthly/quarterly/yearly basis. Usually this report is fairly static, i.e. created once and not changed often. The audit report can include an export of transaction logs with applied filters. For example, an audit report can include customized columns exported and sent to appropriate stakeholders daily, weekly, monthly, etc. This could include sending human resources (HR) an audit report for select departments. The forensics report could be based on an investigation of a specific user, trend, etc. Here, the administrator begins by looking at high level trends then drilling down interactively until relevant information is obtained.
Referring to FIGS. 10-19, in an exemplary embodiment, various screen shots illustrate an exemplary operation of the interactive analytics systems and methods. In particular, FIGS. 10-19 are GUI screens presented to a user of the interactive analytics systems and methods. In an exemplary embodiment, the GUI screens could be presented through a Web browser or the like. FIG. 10 illustrates a first screen shot of the exemplary operation where the user has selected a line graph visualization 1002 and no grouping 1004. As shown in a graph 1006, total transactions are shown within a date range (1/1 12:00 am-1/14 11:49 pm in this example). Since the initial grouping is “no grouping,” the graph 1006 simply displays all transactions in the date range. Once this interaction, i.e. the graph 1006, is created, it is stored in a history list 1010 as a first icon 1012. FIG. 10 illustrates a pull down menu 1020 for the groupings, and in this exemplary embodiment, the user selects URL class from the pull down menu.
FIG. 11 illustrates a resulting graph 1102 from changing the no grouping 1004 to grouping by URL class 1104. Once the graph 1102 is created, it is added as well to the history list 1010 as a second icon 1112. The URL class 1104 shows a grouping of the transactions by different categories of URL class. For example, these can include general browsing, productivity loss, legal liability, etc. These categories are determined by the systems 100, 500 during the monitoring. In FIG. 12, the user selects the legal liability line 1202 in the graph 1102, and a menu 1210 is displayed for further grouping of the legal liability line 1202. In this example, the user selects group by action 1212. FIG. 13 illustrates this grouping of the legal liability line 1202 by action 1212 in a pie chart 1302. Note, in FIG. 13, the visualization is changed to a pie chart 1304 as the grouping by action displays a breakdown of the actions taken on the legal liability line 1202 by the system 100, 500. The action can include allowing or blocking the traffic, and in this example, the traffic was allowed 71% of the time and blocked 29% of the time. Note, the pie chart 1302 is added to the history list 1010 as a third icon 1312. Also in FIG. 13, the user is shown selecting the blocked portion 1320 of the pie chart 1302, and is presented with a menu 1330 for further grouping of the blocked portion 1320 in which the user selects group by user 1340. Filters 1350 are displayed on the side showing an applied filter 1352 of URL class by legal liability.
FIG. 14 illustrates the group by user 1340 of the blocked portion 1320 of the liability line 1202 in a bar chart 1402. The group by user 1340 requires bar chart visualization 1404 (or a list, table, etc.). The bar chart 1402 is added to the history list 1010 as a fourth icon 1412, and a filter 1452 is added for the blocked action. The bar chart 1402 shows transactions by user that were blocked and were legal liability in the URL class. In FIG. 15, the user selects the third icon 1312 in the history list 1010, and the pie chart 1302 is brought back up. In FIG. 15, the user now selects the allowed portion 1502 (instead of the blocked portion 1320). A grouping menu 1504 is brought up for the allowed portion 1502, and the user selects a group by user 1510. FIG. 16 refers again to the bar chart 1402 of FIG. 14. Here, a first user 1602 is selected (abc@zscaler.com), and a menu 1604 is shown and group by domain 1610 is selected.
FIG. 17 illustrates a bar chart 1702 which shows domains for the user 1602. That is, block URLs that are legal liability for the user 1602 during the date range. FIG. 17 also includes a new filter 1752 for the user (abc@zscaler.com). The user selects one domain 1760 (thepiratebay.se), and a menu 1770 is shown from which the user can select transaction details 1780. Also, a fifth icon 1790 is displayed in the history list 1010 for the bar chart 1702. FIG. 18 illustrates a transactions table 1802 of the transaction details 1780 from FIG. 17. A sixth icon 1804 is included in the history list 1010, and the visualization is changed to a transaction table 1806 to display the transaction details 1780. Also, new filters 1810, 1812, 1814 are added to sort through the transaction details 1780. Here, the user is looking for transaction on the domain 1760 (thepiratebay.se) which are greater than 100 Mb and are PDF files. FIG. 19 illustrates an exemplary final report which can be produced from any of the interactions in the history list 1010.
Referring to FIGS. 20-28, in an exemplary embodiment, various screen shots illustrate another exemplary operation of the interactive analytics systems and methods. In particular, FIGS. 20-28 are GUI screens presented to a user of the interactive analytics systems and methods. In an exemplary embodiment, the GUI screens could be presented through a Web browser or the like. FIG. 20 illustrates a first screen shot of the exemplary operation where the user has selected a line graph visualization 2002 and no grouping 2004. FIG. 20 is a line graph 2006 of total transactions for the previous month. FIG. 20 also includes a history list 2010 with a first icon 2012 for the line graph 2006. In FIG. 21, the grouping is changed to a grouping by URL class 2104 which separates the transactions into general surfing, security risk, and business use. FIG. 21 includes a line graph 2106 of the different URL classes, and once created, a second icon 2112 is added to the history list 2010.
Note, the definition for URL class is based on the systems 100, 500 and these definitions could also be modified, customized, etc. For example, in FIGS. 10-19, the URL classes were general browsing, legal liability, and productivity loss. In FIGS. 20-28, the URL classes are general surfing, security risk, and business use. The interactive analytics systems and methods contemplate any number of different classes as defined in the systems 100, 500. Those of ordinary skill in the art will recognize the various groupings described herein can include any number of different categories as appropriate for the systems 100, 500.
In FIG. 22, the general surfing line is selected in the line graph 2106 and a menu 2202 is shown in which a user selects grouping by action 2204. FIG. 23 shows a pie chart 2302 based on the grouping by action 2204 in FIG. 22. For the grouping by action 2204, the visualization in FIG. 23 is changed to a pie chart visualization 2304, and a filter 2306 is added for URL class: general surfing. Also, the history list 2010 now includes a third icon 2310 and a fourth icon 2312 for the previous charts. In FIG. 23, the user can select the allowed transactions for a filter and group by user 2402 which is shown in FIG. 24. FIG. 24 includes a bar chart 2404 with a bar chart visualization 2406 of allowed transactions for general surfing by user. FIG. 24 also includes another icon 2410 in the history list 2010 for the bar chart 2404.
In FIG. 25, the user selects the icon 2312 in the history list 2010, and the pie chart 2302 is again displayed. The user can delete the pie chart 2302 by selecting a delete button on the icon 2312. Upon deleting the pie chart 2302, FIG. 26 illustrates the bar chart 2404 with the icon 2312 removed from the history list. Also, FIG. 26 is an updated bar chart 2602 with a filter 2604 showing allowed transactions by user. FIG. 27 shows an independent example of grouping by domain 2702 and an associated bar chart 2704. FIG. 27 includes no filters and a new icon 2710 in the history list 2010. FIG. 28 shows a new transaction table visualization 2802 of a table 2804 of transactions by users for general surfing that are blocked for a specific domain (Mozilla). Again, a new icon 2810 is added to the history list 2010 for the table 2804.
Referring to FIG. 29, in an exemplary embodiment, a flowchart illustrates an interactive email analytics method 2900. The method 2900 can be implemented with the systems 100, 500, 700 and with the methods 600, 800, 900. For example, the system 700 can be a subset of the systems 100, 500, and the systems 100, 500 can use the method 2900 to collect data about emails. The method 2900 provides visualization of email data collected by the method 600 and stored in the logs 704 of the system 700. That is, the method 2900 can be used for forensic and/or analytic investigations to mine enormous amounts of email data in the logs 704. The steps shown in FIG. 29 for the method 2900 can be executed independently and concurrently as appropriate and in any order. The method 2900 includes selecting a date range (step 2902). The date range can include the current day, current week, current month, previous day, previous week, previous month, or any custom range at any granularity (days, hours, minutes, etc.). The method 2900 includes selecting a visualization type (step 2904). For email analytics, the visualization is a bar chart, a pie chart, a line chart, a totals lists, a summary table, and a transaction table with three view types—transaction View, Message History View, and Administrative Quarantine View.
The method 2900 includes selecting a grouping (step 2906). The email groupings can include action, direction, user, domain, department, email type, etc. With the date range, the visualization type, the data units, and the grouping selected, the method 2900 creates the interaction, displays the interaction, and stores the interaction in a history list (step 2908). Specifically, based on the selections, the method 2900 obtains the requisite data from the logs 704, i.e. based on the date, and grouping, and displays the interaction based on the visualization type. The method 2900 also stores the interaction in a history list that stores a record of the order of the user's investigation. Once created, the user can change any of the aforementioned selections (step 2910). If there is a change, the method 2900 updates the interaction based on the change and stores the updated interaction in the history list (step 2912).
Once the interaction is finalized and presented to the user, the user can modify the presentation such as adding/removing a filter in the interaction (step 2914). The filters can include email filters for direction, action, email type, etc. and email transaction filters for direction, action, email type, server response search, includes attachment, is failed delivery, spam score, message bytes, is retry, department, threat search, transaction ID search, message ID search, sender search, subject search, recipient search, destination IP, source IP, user, reason, domain, DLP dictionary, DLP engine, etc. Also, the method 900 or similar can be implemented with the method 2900 for moving up/down the history list.
The transaction view can simply show a chronological ordering of email transactions that took place. More details can be viewed for each transaction such as via clicking on a transaction and selecting data in a menu. The Message History View shows a list of emails that have been processed by the systems 100, 500 and shows a logical flow (e.g., the message was received, then it was quarantined, and finally the user released the message from quarantine. The purpose of this view is to trace an email. More details can be seen for each message and for messages that have been quarantined; the admin can take actions to release or drop the message from quarantine. The Administrative Quarantine View simply shows all messages that have been quarantined for an organization. More details can be seen for each message and the admin can take actions to release or drop a message from quarantine.
Note, the interactive analytics have been described herein with respect to Internet traffic and email traffic, but those of ordinary skill in the art will appreciate these same concepts can be applied to any type of traffic (e.g., mobile device traffic) since the same workflow concepts apply. The interactive analytics described herein have several advantages. First, due to the cloud based nature of the systems 100, 500, the interactive analytics provide monitoring independent of location, operating system, platform, etc. Thus, all users for an organization can be monitored all of the time regardless of device, method of network connectivity, etc. The interactive analytics includes a history trail of charts investigated to preserve the workflow. This allows an administrator to back to a step in the workflow and change the path of the investigation from any step in the workflow without having to start from scratch. Lastly, documenting the steps in the workflow by exporting it to PDF as this will allow the admin to share the details perhaps with IT management, or the HR department, etc.
1. A computer implemented interactive analytics method, comprising:
logging data traffic between a plurality of users and an external network in a log system, wherein the logging is performed by a cloud based system;
receiving a first request for interactive analytics of the logged data traffic, wherein the first request comprises a date range, a visualization type, and a grouping;
obtaining data traffic from the log system responsive to the first request;
formatting the obtained data traffic based on the date range, the visualization type, and the grouping to provide a first interaction based on the first request;
displaying the first interaction;
storing the first interaction in a history list;
monitoring the data traffic between the plurality of users and the external network via the cloud based system, wherein the monitoring is independent of location, device type, and operating system of each of the plurality of users;
receiving a plurality of additional requests for interactive analytics of the logged data traffic, obtaining data traffic from the log system responsive to the each of the plurality of additional requests, formatting the obtained data traffic to provide a plurality of additional interactions each based on the plurality of additional requests, and storing the plurality of additional interactions in the history list; and
receiving a selection of any interaction in the history list, displaying the selection, receiving a request to modify the selection, and if the selection is not the last interaction in the history list, deleting all subsequent interactions in the history list based on the modified selection.
monitoring Hypertext Transfer Protocol, Hypertext Transfer Protocol Secure, and email traffic associated with the plurality of users.
compressing the monitored data traffic using a plurality of compression techniques prior to logging the data traffic in the log system.
receiving a second request for interactive analytics of the logged data traffic, wherein the second request comprises a modification of the first request;
obtaining data traffic from the log system responsive to the second request;
formatting the obtained data traffic based on the modification to provide a second interaction based on the second request;
displaying the second interaction;
storing the second interaction in the history list.
receiving a selection of any interaction in the history list;
displaying the selection;
receiving an additional request for interactive analytics of the logged data traffic, wherein the additional request comprises a modification of the selection;
obtaining data traffic from the log system responsive to the additional request;
formatting the obtained data traffic to provide the additional request.
6. The method of claim 1, wherein the data traffic comprises Hypertext Transfer Protocol or Hypertext Transfer Protocol Secure traffic;
wherein the visualization type comprises one of a bar chart, a pie chart, a line chart, a totals lists, a summary table, and a table of transactions; and
wherein the grouping comprises one of Application Class, Threat Class, Basic Threat Class, Basic Threat Type, Data Leakage Prevention (DLP) Class, DLP Dictionary, DLP Engine, Instant Message (IM) Direction, Location, Social Direction, Department, User Direction, Uniform Resource Locator (URL) Class, URL Category, URL Super-Category, Action, Web Application, Webmail Direction, and Domain.
7. The method of claim 1, wherein the data traffic comprises email traffic;
wherein the visualization type comprises a table of transactions; and
wherein the grouping comprises any of action, direction, user, domain, department, and email type.
8. A computer implemented interactive analytics method, comprising:
receiving a plurality of additional requests for interactive analytics of the logged data traffic, obtaining data traffic from the log system responsive to the each of the plurality of additional requests, formatting the obtained data traffic to provide a plurality of additional interactions each based on the plurality of additional requests, and storing the plurality of additional interactions in the history list;
wherein the data traffic comprises email traffic;
displaying the selection, receiving a request to modify the selection; and
if the selection is not the last interaction in the history list, deleting all subsequent interactions in the history list based on the modified selection.
14. The method of claim 8, wherein the data traffic comprises Hypertext Transfer Protocol or Hypertext Transfer Protocol Secure traffic;
US13/495,537 2012-06-13 2012-06-13 Systems and methods for interactive analytics of internet traffic Active 2033-05-28 US9003023B2 (en)
US13/495,537 US9003023B2 (en) 2012-06-13 2012-06-13 Systems and methods for interactive analytics of internet traffic
US20130339514A1 US20130339514A1 (en) 2013-12-19
US9003023B2 true US9003023B2 (en) 2015-04-07
ID=49756972
US13/495,537 Active 2033-05-28 US9003023B2 (en) 2012-06-13 2012-06-13 Systems and methods for interactive analytics of internet traffic
US (1) US9003023B2 (en)
US10365998B2 (en) * 2015-09-14 2019-07-30 International Business Machines Corporation Modifying monitoring configurations that support analytics programs
2012-06-13 US US13/495,537 patent/US9003023B2/en active Active
US20130339514A1 (en) 2013-12-19
Free format text: ASSIGNMENT OF ASSIGNORS INTEREST;ASSIGNORS:CRANK, SAMUEL JOHN;HAZARIKA, UPASONA KATH;SIGNING DATES FROM 20120607 TO 20120613;REEL/FRAME:028369/0014