IDENTIFYING CHANGES IN ON-LINE DATA REPOSITORIES

Abstract: A system for accessing documents contained in a remote repository, which change in content from version-to-version. The system allows users to specify lists of documents of interest. Based on the lists, the system maintains an archive, which contains a copy of one version of each listed document, and material from which the other versions can be reconstructed. The system periodically compares the archive with current versions of the documents located in the respository, and updates the archive, thereby maintaining the ability to reconstruct current versions. The system also monitors access to the versions by each user. When a user calls for a current version, the system presents the current version, and indicates what parts of the current version have not been previously accessed by the user.

Description: IDENTIFYING CHANGES IN ON-LINE DATA REPOSITORIES The invention concerns presentation of a current version of a document retrieved from a data repository. The presentation indicates changes made in the document since the viewer accessed a previous version. BACKGROUND OF THE INVENTION Information which is stored in computerized systems can change freguently, and without notice. As an example, software under de\elopment freguently involves many persons, and is commonly stored at a central location. Each person can change the software on an ad hoc basis, without knowledge of others. In such systems containing changeable data, a person who examines information on a given day does not, in general, know whether, and how, the information has changed since a previous examination. Conseguently, the person must spend time comparing currently available information with previous versions of the information. Software exists for facilitating this comparison. For example, systems known as "version control systems," or "revision control systems," store data which represents multiple versions of different documents, as indicated in Figure IA. In that Figure, the DATA is indicated, together with dashed loops which indicate the VERSIONS. The loops indicate that the VERSIONS are contained in, and MISSING UPON TIME OF PUBLICATION MISSING UPON TIME OF PUBLICATION interspersed among a much larger document. Small arrows point to changes, which are primarily additions in this case. The change in the "last update" date give an example of text being replaced. Here the page's author had highlighted the changes manually with small icons as well. The banner at the top of the page was inserted by HTMLDIFF. Figure 12 illustrates version histories which give the user a chance to compare any two versions, or to go directly to a selected version. Figure 13 illustrates output of 3NE ER, and shows a number of anchors (the descriptive text originates from the hot list) . The anchors marked "changed" have modification dates after the time which the user's browser history indicates the URL was last seen. Some URLs were not checked at all, and others were checked and are known to have been seen by the user. Figure 14 demonstrates use of a SNAPSHOT facility, which allows a user to specify an operation on a URL. In this example, DOUGLIS@RESEARCH.ATT.COM is "remembering" URL HTTP://SNAPPLE.CS.WASHINGTON.EDU:600/MOBILE/. DETAILED DESCRIPTION OF THE INVENTION A TECHNICAL APPENDIX, which is located at the end of this Specification, describes the invention in detail. Following the TECHNICAL APPENDIX is a COMPUTER PROGRAM LISTING, which contains code which implements one form of the invention. An illustrative embodiment of the invention is given in the discussion below. Overview of Invention A commonly used repository of information is known as the World Wide Web, or WWW. In the WWW, providers of information make their information available to users in the form of "pages." Each page is assigned a name, which distinguishes the page from other pages, and allows a user to locate the page. The WWW provides information using an information retrieval- and-display approach called "hypertext." In hypertext, a page may contain references to other pages, or other documents. A user can call up a pagu which is referenced, by clicking on tie reference (called a URL, or Universal Resource Locator) with a pointing device. Figure IB provides an example. In Figure IB, a document D is displayed to a user. References R refer to other documents. For example, RI refers to Dl, R2 refers to D2 , and so on. The referenced documents themselves may contain their own references to other documents, such as R4, which refers to D4. A user can retrieve a referenced document D, by clicking on the reference R which refers to it. For example, clicking on RI causes retrieval and display of Dl. Under the invention, a user of the WWW initially identifies pages of interest. Document D in Figure IB represents one page. These selected pages form a "hot list." Then, the invention does the following: (a) Copies the hot-listed pages into an archive, which is a storage location separate from the WWW, and under independent control. After the copying, the original pages continue to reside in the WWW, and copies reside in the archive. (b) Monitors, at later times, the original pages for changes, and archives the changes. (c) Records the times when the user later accesses each hot-listed page. (d) Whenever the user accesses a hot-listed page, presents the user with i) the current version of the page (whLCh may differ from the initial copy which was stored in the archive) ; and ii) an option to compare selected versions of the page. The comparison is presented by performing a differencing operation on pairs of versions. e) As an option, the invention also implements the steps described above with respect to documents referenced by the page. For example, in Figure IA, if a user is viewing document D, the invention can present the current version of reference document D2, together with a history of D2. More Detailed Description Hot-List Paσes are Stored in EXTERNAL SERVICE Figure 1 illustrates a REPOSITORY of information, such as the WWW. For assistance in accessing the REPOSITORY, the invention provides the EXTERNAL SERVICE which includes: (a) SOFTWARE, such as that provided in the COMPUTER PROGRAM LISTING herein, (b) a SERVER, or other computer, which runs the software, and (c) COMMUNICATION SYSTEMS which link with both the users and the REPOSITOFY. The SERVER and the COMMUNICATION SYSTEMS located within the EXTERNAL SERVICE are known in the art. As indicated in the Figure, the EXTERNAL SERVICE is distinct from the REPOSITORY, and under separate control. The invention does not disrupt the users' normal interaction with the REPOSITORY; the users can interact with both the REPOSITORY, as usual, and also with the EXTERNAL SERVICE. Dashed arrows 3 indicate the interaction. Several examples will provide illustrative modes of operation of the invention. Example: Single User Operation with respect to a single user will first be explained. Figure 2 shows a hot list 4, submitted by USER 1, which identifies pages A and B as being of interest to USER 1. The invention allows the user to modify the hot list at later times. In response to the hot list, the invention copies pages A and B from -the REPOSITORY, as indicated by the dashed arrows. These PAGEs will be termed "base pages." At this time, the originals of PAGES A and B remain in the REPOSITORY, and copies reside in the EXTERNAL SERVICE. Then, the invention periodically examines the originals of PAGES A and B, located in the REPOSITORY, for changes. In looking for changes, the invention first performs a preliminary check, based on information such as (1) dates of modification and (2) checksums. Dates of mocification may be added to a PAGE by .he PAGE provider. These dates directly indicate whether the originally archived version has changed. Checksums are generated by the invention. An example of a checksum is the numerical sum of all characters in a line, or on a page. If a checksum changes (indicating that the number of characters has changed) , the change indicates a high probability that a change has occurred in the PAGE. (In practice, the checksums used are more complex than this simple example illustrates. Checksums are known in the art.) If the preliminary check, either by dates of modification or checksums, indicates that changes have occurred, then the invention copies the present version of the PAGE into the EXTERNAL SERVICE, and compares it with the base page, in order to locate the changes. Computer programs for detecting such changes are known in the art, and some examples are given in the TECHNICAL APPENDIX. A preferred program, not known in the prior art, is entitled W3NEWER, and was developed by the inventors. W3NEWER is contained in the listing located at the end of this Specification. When changes are found, the invention stores them in the EXTERNAL SERVICE. Figure 3 illustrates storage of the changes, by the small boxes 6 located below PAGEs A and B. The DATEs within the boxes 6 indicate the dates on which the changes were saved. Figure 3A illustrates how the invention displays the history of versions. Column 7 indicates the number assigned to each version by the invention. Column 8 indicates the times when the respective versions were retrieved by the invention. Column 8A allows a user to select a pair of versions for a differencing operation, as discussed below. For ease of explanation, Figure 3 illustrates storage of base pages, which are early versions of PAGEs, together with subsequent changes, indicated by the boxes 6. However, in practice, it can be more efficient to perform storage in a reversed sense, by storing the latest version as the base page (instead of the early version) and storing the changes 6 from which early versions can be reconstructed. One reason is that users are expected to call for latest versions more frequently than early versions. Storage of the entire latest versions eliminates the need to reconstruct them. The changes, together with their base pages, form an archive, which allows reconstruction of a PAGE as of any date desired. For example: PAGE A itself (ie, the base page) , plus the changes labeled DATE 1, allow reconstruction of the version of PAGE A, as of DATE 1. PAGE A itself, plus the changes labeled DATE 1 and DATE 2 , allow reconstruction, as of DATE 2, and so on. When USER 1 wishes to view PAGE A, the invention ordinarily retrieves and presents the current version. The invention also provides an option for reconstructing the PAGE, as of a date specified by the user, and presents it in the format shown in Figure 4. The program HTMLDIFF, contained in the listing, generates the image shown . n Figure 4. The content of the page can be divided into three classes. The first class contains material which has not changed. This class of material is displayed in the font, size, color, and background, as customary in documents downloaded from the REPOSITORY. The second class represents changes, and contains material not present in the base page, but which has been added. Brackets 9 indicate such material. (The brackets 9 are part of Figure 4, and are not necessarily part of the page generated by the invention.) This material is presented in a particular font, particular size, particular color, and particular background. The choice of these parameters can be varied but, in general, they should be chosen to maximize contrast with the first class of material. In addition to the formatting described immediately above, the added material is further highlighted by arrows 7. The third class contains material which was deleted from the base page. Deleted material can be handled in at least three ways. One, deleted material can be simply deleted, so that the page presented to the reader contains no reference to the deleted material. Two, the deleted material can be deleted, but a reference indicating the deletion is added, such as the phrase "Deleted material occurs here." In this case, the user can be given the option of fetching the deleted, non-visible, material. Three, deleted material can be presented, but indicated as •'.eleted, as by "redline" format, in /hich a horizontal line, perhaps red in color, is drawn through the deleted material. Figure 3B illustrates a display, generated by the invention, which indicates which PAGEs on a user's hot list have undergone changes. Second Example: Multiple Users In actual practice, multiple users are expected to use the invention. Each of them submits a hot list. In one approach of the invention, the procedure undertaken for a single user (described above) is repeated for multiple users: all PAGEs, on all hot lists, are copied into the EXTERNAL SERVICE. Then, for each hot list, the originals of the PAGES, located within the REPOSITORY, are monitored for changes, and the changes are retrieved into the EXTERNAL SERVICE, as described above. However, this approach contains inefficiencies. For example, a given PAGE will probably be identified by more than one hot list. Repeatedly copying that PAGE, for each hot list, would entail storage of multiple copies of the same PAGE. Further, repeatedly comparing the multiple copies with their originals in the REPOSITORY represents a waste of computer time: a single comparison would suffice. The invention reduces these inefficiencies by the approach shown in Figure 5. This Figure represents a modification of Figure 4, to which a hot list for USER 2 has been added. The added hot list specifies PAGES A and C. To process the new hot list, the invention first checks whether the PAGEs identified on the added hot list are archived within the EXTERNAL SERVICE. Since PAGE A, plus its changes, are already contained within the archive, that PAGE is not copied. But PAGE C, which is not present in the ARCHIVE, is archived, as indicated by the dashed arrow. At this time, all PAGEs identified on all hot lists are contained within the archive. To emphasize this fact, PAGE A is indicated twice: once for USER 1, and a second time by a dashed page 14, for USER 2, although, as stated above, PAGE A is stored only once. After archiving all necessary PAGEs, the originals, located within the REPOSITORY, are periodically monitored for changes, as described above. The changes are copied to the archive of the EXTERNAL SERVICE. Flow Chart An exemplary flow chart is shown in Figure 6, which refers to a single-user case. In block 20, the EXTERNAL SERVICE accepts hot lists from users. Then, in block 23, the EXTERNAL SERVICE checks whether the PAGEs identified on the hot lists are contained within the archive. If not, the PAGEs are copied from the REPOSITORY, as indicated by block 26. Then the logic proceeds to block 29, where the originals of the PAGEs, located in the REPOSITORY, are examined for changes. The examination can include the preliminary checks (for checksums and dates of modification) discussed above. Wh=n changes are found, the entire PAGE containing them is downloaded to the EXTERNAL SERVICE, and the changes, indicated by blocks 6 in Figure 3, are derived. Block 32 indicates relevant information stored in the EXTERNAL SERVICE. As users access the PAGEs, block 35 monitors the times of the accesses, in order to identify which versions of each PAGE the user viewed last. These times are stored, as indicated by block 32 and dashed arrow 37. These times are used to determine which changes in Figure 4 are to be identified as new material, when a PAGE is called by each user. An example will illustrate. Figure 7, top, illustrates the time-history of changes made to PAGE A. USER 1 accessed this PAGE at time 2, as indicated. Block 35 in Figure 6 monitors and records this time (at TIME 2 in Figure 7, and not earlier, of course) . If USER 1 again accesses the PAGE at time 5, then the invention presents VERSION 1 to the USER. However, if the user accesses the PAGE at time 11, VERSION 2 had been created since the last access by USER 1. The invention had previously identified the changes, and copied them as indicated in Figure 3. Now, at the access at time 11, the invention presents VERSION 1, plus the changes which make VERSION 2, because block 35 in Figure 6 indicates that the USER has not seen VERSION 2. Returning to the flow chart of Figure 6, block 39 indicates that, when a USER calls for a PAGE, the invention presents the current version, and indicates the changes made (as in Figure 4) since the USER last accessed that page. In th» example immediately above, the invention presents VERSION 2 of PAGE A, as in Figure 7, and indicates the changes made since VERSION 1, because VERSION 1 was the last accessed by USER 1. The flow chart of Figure 6 should not be read as limiting the invention to a linear, sequential mode of operation. In practice, multiple users can present hot lists simultaneously, and other operations shown in the flow chart can also occur together. Third Example: Notification of Changes The invention can notify USERs when changes in their hot- listed PAGEs occur, as indicated by the dashed block 40 in Figure 6. This notification can take the form of a flag which is associated with the BASE PAGE in Figure 8. When the USER logs into the EXTERNAL SERVICE, the invention notifies the USER of the changes to the respective PAGEs. Figure 3B illustrates one approach to identifying PAGEs which have changed. .Other types of notification are possible. For example, the invention need not wait for a user to access a PAGE. The invention can notify the user when changes have been found, as by sending an electronic mail message to the user. Fourth Example: Common Hot List The invention can maintain a predetermined hot-list, for a community of users. This hot list contains a list of PAGEs which are considered to be of general interest to the community. This hot list, and the PAGEs identified on it, are made publicly available, to all users, but on a read-only basis. Users cannot modify the hot list, or the pages. This predetermined hot list can serve as an instructional tool, to educate users in the operation of the invention, and to demonstrate desirable features. One Architecture of Data Storage An illustrative approach to storage of the information identified in block 32 of the flow chart of Figure 6 is illustrated in Figure 8, which is explained with reference to Figure 7. Figure 7 illustrates hypothetical changes to the three PAGEs identified by the two hot lists of Figure 5. PAGE A underwent changes at times 7 and 13. Page B underwent changes at time 10, and so on. In Figure 8, the arrows extending from the symbols "USER 1", etc., indicate the times of access by the users. For example, USER 1 accessed PAGE A, VERSION 1, at time 2. USER 1 then accessed PAGE A, VERSION 2, at time 9, and so on. The invention maintains a TABLE of these times, as indicated on the right side of Figure 8, together with a list of PAGEs, or documents, owned by each USER. ' Ownership is determined by the hot lists. The invention also maintains (a) the BASE PAGES, (b) the changes to each, and (c) the times of each change, as indicated on the left side of the Figure. From this data, the invention is able to reconstruct any PAGE, as of any date subsequent to the date of the BASE PAGE. Additional Considerations 1. One definition of "page" is that it refers to a unit of data, stored in a system, which is identified by a specific name. (In the WWW, all pages have unique names.) Other terms can refer to such units of data, such as "files" and "documents." In general, the particular name used will depend on the system storing the data. 2. One definition of "repository" is a collection of data, which is accessible by computer. The repository may be available to the public, or access may be limited. In general, repositories are expected to be distributed, meaning that the storage locations are physically distributed over a wide geographic area, and linked together by a communication system. 3. It was stated above that the invention can reconstruct a page as of any selected date. The reconstruction is based on the changes 6 in Figure 3. These changes are detected periodically, and the periodicity is determined by each user of the system, subject to limits imposed by the designer and system administrator. For example, user A can specify a period of one day for checking for changes in the pages on user A's hot list; user B can specify a different period for B's pages, such as one week. The system administrator can specify that no period, for any user, can be shorter than one hour. Consequently, changes in a page, located in the REPOSITORY, will only appear in a reconstruction done by the EXTERNAL SERVICE after the changes have been detected, and not earlier. An example will illustrate this distinction. Assume that the invention looks for changes on odd-numbered dates. Thus, a change occurring on the fourth of a month will be detected on the fifth. However, if a user happens to call for reconstruction on the fourth, the change occurring on the fourth will not appear in the reconstruction. Only changes occurring as of the prior detection, namely, as of the third, will appear. It is expected that the detection process will be performed sufficiently often that the influence of this factor will be negligible. 4. The invention can extend its differencing function (ie, the examination of pages for changes) to pages referenced by the page .accessed by the user. For example, if the user accesses document D in Figure IB, the invention can detect changes in all documents referenced by document D, such as Dl, D2, and D3. In another embodiment, the differencing can extend to the documents which are, in turn, referenced by the referenced documents. For example, the referenced documents (Dl, D2, and D3) refer to D5 and D6. These latter documents (D5 and D6) can be differenced also, as can be the documents which they reference, and so on. 5. The invention provides all information from which a current version of a PAGE may be derived. Figure 4 gives an example. Figure 4 contains all such information, together with other information which indicates changes since a previous version. 6. The discussion above presumed that comparison, or differencing, between different versions of a PAGE was done within the EXTERNAL SERVICE. This is not strictly necessary; the comparison can be done at any convenient location. Further, the preliminary checking for the existence of changes can be done at any convenient location. 7. In data storage systems, names are given to the units of information (e.g., documents, pages, records), although the names can be different in different databases. However, the names of the units, in general, remain the same throughout time, despite changes which, are made to the information contained in the unit. Therefore, one definition of the term "version" refers to a unit of information, which is different from a previous unit of the same name. 8. The REPOSITORY in Figure 1 is, in general, located remotely from the EXTERNAL SERVICE. Communication is undertaken by any convenient approach, such as a public-access communication network known as the INTERNET. In general, the REPOSITORY is undtr independent control of the EXTERNAL SERVICE. One ramification of this independent control is that the type of processing done to the PAGEs copied into the EXTERNAL SERVICE is controlled by the EXTERNAL SERVICE, and not by the REPOSITORY. For example, (a) the particular processes used in locating and storing differences, (b)- the frequency of processing, and (c) the mode of notifying a user, are controlled by the designer of the EXTERNAL SERVICE. The operator of the REPOSITORY has no involvement in this processing. 9. Figure 9 illustrates another form of the invention. The invention maintains base pages 30 within the EXTERNAL SERVICE, as required by the hot lists 36. The base pages 30 were downloaded from respective repositories 42A, 42B, etc. The invention periodically monitors the originals 30A of the pages, located in the repository 42, for changes, and stores the changes within the EXTERNAL SERVICE. The invention notifies users when changes are found in pages on their hot lists (notification is not shown) . A version control system 39 allows users to fetch and view any version of any page. 10. The different versions of documents may contain drawings, files from which sound may be generated, files which produce video clips and animation, and other components which do not consist strictly of alphanumeric characters. The invention detects the existence of changes in such components, and marks the existence of the changes, in the display as shown in Figure 4, without necessarily identifying in detail the nature of the changes. 11. A primary use of the invention is envisioned in the situation shown in Figure 10. The EXTERNAL SERVICE obtains copies of PAGEs from a REPOSITORY, such as WWW. However, the EXTERNAL SERVICE is given no authority to replace or modify the pages contained in the REPOSITORY. To the EXTERNAL SERVICE, the PAGEs represent read-only data, as indicated by the "X" over arrow 50, which indicates a write operation. The EXTERNAL SERVICE performs differencing between currently copied versions of pages, and DATA representing previous versions. The DATA stored in the EXTERNAL SERVICE can be both read, and written to, by the EXTERNAL SERVICE. The EXTERNAL SERVICE reconstructs any version on demand, and also indicates differences between any two versions selected by a user, as discussed above. These. functions can be accomplished by a prior-art Revision Control System, RCS (also called a Version Control System) , or by the code contained in the listing contained in this Specification. 12. In one form of the invention, the PAGEs retrieved are written in a "markup language," such as HyperText Mark-up Language (HTML). A mark-up language, in general, contains two types of codes, interspersed among the actual text of a document. One type indicates how the PAGEs are to be displayed. For example, some codes indicate paragraph indentation, ov ' .ier codes indicate font styles, yet other codes indicate style of font, within a font, such as italicizing, underlining, double-striking, or bold printing. This type of code is referred to as format- defining. A second type of code can identify an image, such as a bit¬ mapped file located elsewhere. When such a code is read by the system displaying the PAGE, a copy of the image is retrieved, and displayed within the PAGE, at the location specified by the code. This type of code is referred to as content-defining. The invention does not treat changes in the format-defining codes as changes in content. Thus, a PAGE which changes in layout, or typestyle, only, is not designated as a changed page. The differencing program contained in the COMPUTER PROGRAM LISTING compares different versions on a subunit-by-subunit basis. For example, the program compares corresponding sentences in different versions, and the sentences are detected by sentence terminators. (Longer subunits can be used, such as paragraphs or pages.) The sentence terminators are a subset of the markup language. Specifically, the terminators are format-defining codes. COMPUTER PROGRAM LISTING The program listing is divided into three sections. 1. HTMLDIFF, comprising: html_diff. sml (5 pages) , diff.s l (3 pages) , mlweb.sml (4 pages) , and html. lex (one page) . 2. W3NEWER (17 pages) . 3. NOHANDS, comprising: nohandsBE (11 pages) , no-hands.cgi (3 pages), rcsdiff.cgo (4 pages) , and snapshot.cgi (3 pages) . NOHANDS is an overall program set which utilizes W3NEWER and HTMLDIFF. TECHNICAL APPENDIX A TECHNICAL APPENDIX, totalling 12 pages, two of which are blank, follows, and refers to Figures 11 - 14 Numerous substitutions and modifications can be undertaken without departing from the true spirit and scope of the invention. What is desired to be secured by Letters Patent is the invention as defined in the following claims.

Claims: CLAIMS 1. A system, comprising: a) means for copying versions of pages from an external repository, which provides pages to the system on a read¬ only basis; b) means for storing data from which i) selected versions can be reconstructed; and ii) differences between selected versions can be identified. 2. A system according to claim 1, in which the versions reconstructed are specified on lists provided by users. 3. In a system for allowing multiple users to interface with another system which contains versions of data, at least some of which change as time progresses, the improvement comprising: a) means for allowing a user to identify a current version and a previous version; and b) means for displaying the current version in a format which distinguishes the current version from the previous version. 4. A method of communicating with a repository which stores data units in the form of current versions which change over time, comprising: a) accepting lists of data units from multiple users; b) maintaining an archive, which contains material which allows reconstruction of selected versions of data units contained on the lists; and c) at intervals, comparing current versions with archived material, and updating the archive. 5. Method according to claim 4, and further comprising the step of d) recording which versions each user has accessed. 6. Method according to claim 5, and further comprising the steps of: e) accepting a request from a user for a data unit; f) presenting a current version of the data unit; and g) highlighting material within the current version which was absent from previous versions accessed by the user. 7. A method of communicating with a repository which stores data units in the form of versions which change over time, comprising: a) maintaining an archive, which includes material which allows reconstruction of selected versions of data units; b) accepting lists of data units from users; c) for each list accepted, checking the archive and, if listed data units are not archived, copying a current version from the repository; d) at intervals, i) finding differences between (A) versions contained in the archive, and (B) current versions, contained in the repository, and ii) storing the differences in the archive; e) maintaining information about each user, which indicates most recent versions accessed by the user; and f) accepting a request from a user for a curi ≤nt version and, in response, i) copying the current version from the repository; ii) finding differences between the current version and the most recent versions accessed; and iii) displaying the current version in a format which highlights the differences. 8. Apparatus for comparing a first text written in a markup language, which includes sequences of words and markup commands, with a second such text comprising: a) means for comparing the first and second text, which compares sequences in the first and second text which are defined by a group of terminators, including sentence-ending punctuation and markup commands, belonging to a first subset thereof, and produces a result which indicates differences between the texts; and b) means for receiving the result and displaying the differences in response thereto. 9. Apparatus for comparing versions of pages in a repository, comprising: a) means for detecting that a second page is a new version of a first page; b) means responsive to the detecting means for comparing the first page with the second page to produce a result which indicates differences between the first and second pages; and c) means responsive to the result for displaying the differences. 10. A method for detecting whether a page in a repository has been modified since it was last examined by a given user, the method comprising the steps performed in the computer system used by the given user of: a) maintaining a first record of last viewed times for pages in that computer system which indicates, for each page in a first set of pages, the last time at which the given user viewed the page and for a given page for which a last viewed time is recorded in the first record, performing the steps of obtaining a last modified time for the given page from a second record in that computer system of last modified times of pages and determining whether the last modified time from the second record is a time such that the time is later than the last viewed time for the given record; b) if the second record does not provide a last modified time which is such a time, obtaining a last modified time from a source external to that computer system to which the user has access and determining whether the last modified time from the external source is such a tiiu..; and c) if the last modified time obtained from the external source is such a time, updating the second record with that last modified time. 11. A method for maintaining a user-defined version history of a page, the method comprising the steps of: a) receiving an indication from the user specifying that a given version of a page is to be retained in a version history for that page; b) if the given version is the first version of the page in the version history, storing a copy of the given version in the version history; and c) otherwise storing at least any difference between the given version and the most recent previous version in the version history. TECHNICAL APPENDIX Tracking and Viewing Changes on the Web Abstract Other pages may be ignored, or browsed by the user only to find they have not changed. We describe a set of tools thai detect when World-Wide- In recent months, several tools have become available Web pages have been modified and present the modifica¬ to address the problem of determining when a page has tions visually to the user through marked-up HTML. The changed. One example of such a tool is webwatch, a prod¬ tools consist of three components: w3newer, which detects uct for Windows thai uses the HTTP HEAD command to changes to pages; snapshot, which permits a user to store find out when a page has been modified since u was last a copy of an arbitrary Web page and to compare any subse¬ viewed by a user's web browser, and generates a repon in quent version of a page with the saved version; and htmldijf, HTML that allows the user to go directly to those updated which marks up HTML text to indicate how it has changed pages. Another example is w3new, by Brooks Cutter, a from a previous version. We refer to the tools collectively public-domain peri script that runs on UNIX® [2]. as the Network-Oriented HTML Archival, Notification, and Each of these tools suffers from a significant deficiency: Differencing System (NO HANDS). This paper discusses while they provide the user with the knowledge that the several aspects of NO HANDS, with an emphasis on systems page has changed, they do not show how the page has issues such as scalability, security, and error conditions. changed. Although a few pages are edited by their main- tainers to highlight the most recent changes, often the modi¬ fications are not prominent, especially if the pages are large. 1 Introduction Even pages with special highlighting of recent changes are problematic: if a user visits a page frequently, what is Use of the World-Wide-Web (W 3 ) has increased dramati¬ "new" to the maintainer may not be "new" to the user. Al¬ cally over the past couple of years, both in the volume of ternatively, a user who visits a page infrequently may m ss traffic and the variety of users and content providers. The changes thai the maintainer deems to be old. W 3 has become an informaiion distribution medium for We have developed a system that efficiency tracks when academic environments (its original motivation), commer¬ pages change, compactly stores versions on a per-user cial ones, and virtual communities of people who share in¬ basis, and automatically compares and presents the dif¬ terests in a wide variety of topics. Information that used ferences between pages. NO HANDS (.Network-Oriented to be sent out over electromc mail or USENET, both active HTML Archival, Notification, and Differencing System) media that go to users who have subscribed to mailing lists provides "personalized" views of versions of W 3 pages or newsgroups, can now be posted on a W 3 page. Users with three tools. The first, w3 newer, is a more scalable ver¬ interested in thai data then visit the page to get the new in¬ sion of Cutter's w3new modification tracking tool that pe¬ formation. riodically accesses the W 3 to find when pages on a user's The URLs of pages of interest to a user can be saved in a hotlist have changed. The second, snapshot, allows a user "hodist" (known as a bookmark file in Netscape ), so they to save versions of a page and later use a third tool, htmldiff, can be visited conveniently. How does a user Jάnd out when to see how it has changed. Htmldiff automatically compares pages have changed? If users know thai pages contain up- two HTML pages and creates a "merged" age to show the to-ihe-minuiedata (such as stock quotes), or are frequently differences with special HTML markups. changed by their owners, they may visit the pages often. Windows is a trademark of Microsoft. Netscape is t trademark of Netscape Corπ unicauons Corporation. ® UNIX is a registered trademark of X/Open. Tracking and Viewing Changes on the Web (NO HANDS) checked The threshold can vary depending on the URL. tency. In the case of pages that are ot interest to a user _>u: e aire-cv re me.r- -.. sr could be leu :o a ie and the center.. notifies the repository Either clients polling about b> Gweπz- the o: take xtions tnougn unified report supports the 'file changed on the user s would allow individual users to take snapsnots of files ma; 2.3 Cache Consistency Issues are not already under the control of a versioning system such as the Revision Control System (RCS) [14]; this might Determining when HTTP pages have changed is analogous be an appropnate use of a browser with client-side execu¬ to cxhing a file in a distributed file system and determin¬ tion, such as HoUava [13]. ing when the file has been modified. While file systems such as the Andrew File System [6] and Spnte [8] provide guarantees of cache consistency by issuing call-backs to 2.4 Error Conditions nosts with invalid copies. HTTP access is closer to the tradi¬ tional NFS [12] approxh, in which clients check bxk with When a periodic task checks the status of a large πumoer servers periodically for exh file they access. Netscape can of URLs, a number of things can go wrong Local DΓOC- be configured to check the modification date of a cached lems such as network connectivity or the status of a prox> - page exh time it is visited, once exh session, or not at all cxhmg server can cause all HTTP requests to fail Prox> - Cxhing servers check when a client forces a full reload, or cxhing servers are sometimes overloaded to me point ct after a ume-to-live value expires. timing out large numbers of requests, and a bxkground task Here the problem is comphcaied by the target environ¬ that retrieves many URLs in ashoπ time can aggravate tne:r ment, one wishes to know not only when a currently viewed condiuon W3newer should therefore be able to αetec: cases page has changes, but also wnen a page that has not been when it should aboπ and try again later (.preferaDh in time seen in a while has changed. Fortunately, unlike with file for the user to see an updated report) systems. HTTP daia can usually tolerate some inconsis¬ At the same time, a number of errors can arise v-itn ιnαι Trdcλ-in-r and Viewmz Changes on the Web (NO HANDS) vidual URLs. They can move, with or withouileaving a for¬ cess multiple versions of a page for the purposes oi compar¬ warding pointer. The server tor a URL can be dexti vated or ison. This section descπbes methods for maintaining ver¬ renamed. They may disallow retneval by "robots." mean¬ sion histoπes and several issues that anse v. itn our solution ing that anyprogram that follows the "robot exclusion pro¬ tocol" [10] will not retrieve them. Since the cost of retπev¬ ing modification dates is small m comparison to the cost 3.1 Alternative Approaches of retrieving robots txi (pan of the exclusion protocol), it There are three possible approxhes for may well be appropπate to ignore the robot exclusion pro¬ mg of W 3 pages: making each content provider keeo a nis- tocol for this task, or to check robots txt oniy occasionally tory of all versions, making exh user keep mis nistory. or on exh host. Observing the protocol will still be advisable stoπng the version histoπes on an external server. for hosts on which many URLs are checked, especially if the pages' contents are retπeved exh time. Server-side support Exh server could store a p.i.:or> o; Finally, automatic detection of modifications oased on in¬ us pages and provide a mechanism to use ma; .-.i.torv formation such as modification date and checksum can lead to produce marked-up pages that hignhgn; :.-.ar.ges to the generation of "juru mail" as "noisy" modifications This method requires arbitrary content ια:s to tπgger cnange notifications. For instance, pages that re¬ provide versioning and differencing, so it is net rrxti- port tne numoer of times they have been xcessed. or em¬ cal. althougn it is desirable to suppoπ mis feature w nen bed tne current time, w ill look different every time they are the content provider is willing. (See Section 6 . retπeved Client-side support Exh user could run a program that W3ne\ver attempts to address these issues by the follow¬ would store items in the hothst locally, one run html¬ ing steps diff against a locally sav. j copy This metr.od re¬ • If a URL is mxct vible to robots, that fxi is cxhed quires that every page of merest be saved even so the page is not xcessed again unless a special flag user, which is unaitrxuve as the numoer ot pages in is set when the scnpt is invoked. the average user's hothst increases, and it also reαuires the ability to run htmldiff on every platform mat runs • Another flag can tell w3newer to treat error condi¬ a W 3 browser. Stoπng the pages referenced b> the tions as a successful check as far as the URL's times- hothst may not be too unreasonable, since programs ta p goes. For instance, if w3newer runs daily and like Netscape may cache pages locally anywa> There checks a paπicular URL every four days, normally an are other external tools such as wαrmlist [ 16] that pro¬ error xcessing the page on Monday will cause it to be vide this functionality. checked again on Tuesday. With this flag, it would be checked again on Fπday. In general, it seems thai er¬ External service Our approxh is to run a service that is rors are likely to be transient, and checking the next separate from both the content provider and the client time w3newer is run would be reasonable. Pages can be registered with the service via an HTML • When a URL is mxcessible. an error message appears form, and differences can be retnevec in the same in the status repoπ. so the user can take xuon to re¬ fashion. Once a page is stored with the service, subse¬ move a URL that no longer exists or repeatedly hits quent requests to remember the state of the page result errors. in an RCS "check-in" operauon that saves only the dif¬ ferences between the page and its previously checked- In addition. winewer could be modified to keep a running m version. Thus, except forpages that cnange in many counter of the number of times an error is encountered for respects at once, the storage overhead is minimal be¬ a paπicular URL. or to skip subsequent URLs for a host if yond the need to save a copy of the page in tne first a host or network error (such as "timeout" or "network un- plxe. rexhable") has already occurred. Addressing the problem ot "noisy" modifications will require heuπstics to examine Drawbxks to the "external service" approxh are mat the the differences at a semantic level service must remember the state of every page mat anyone who uses the service has indicated an interest in ar.a must 3 Snapshots: External Representa¬ know which user has seen which version of exh page The first issue is pπmanly one of resource allocation, ar.c is not tions of Version Histories expected to be a significant issue unless me service :s used by a great many clients on a number of large pages The In j mon to providing a mechanism for determining when second issue is addressed by using RCS's suppor: :cr dat- H 'J i _ges have been modified, there must be a way to x- estamps and requesting a page as it existed at a particular Tracking and Viewing Changes on the Web (NO HANDS) time a version number could oe retained for htmldiff on the server can result high rocessor .cad. exh <user URL> combination it me fxihty is heavily used These loads can ce Relative links become a problem when a page is moved ated by cxmng the output ot htmldiff lor a uniie. s away from the mxhine that oπg ally provided it. If users who have seen versions .V and V - 1 o: a page cou.d the source were passed along unmodified, then the W 3 retneve htmldiff (pag s , pages -- . ) with a single invoca¬ browser would consider links to be relative to the CGI di¬ tion of htmldiff. The fxi ty could also lmnose a limit on rectory containing the snapshot scπpt. HTML supports a the number of simultaneous users, or replicate itseif among BASE directive that makes relative links relative to a differ¬ multiple computers, as many W 3 service ent URL. which mostly addresses this problem: however. Disk spxe is potentially a problem if e repositon can Netscape 1 IN treats intemal links within such a document grow without bound and with no cost to us users Ir. tact. to be relative to the new BASE as well, which can cause the before a service like this could be plxed on the Internet, it browser to jump between the htmldiff output and the oπgi- would have to authenticate exh user and limit the user to a nal document unexpectedly fixed number of URLs and or disk blocks Most one would use an Internet commerce txility to cnarge α tee in excnange tor permission to store a collection of URLs, this 3.2 System Issues fee could easily offset the cost of the storage medium since it would also be paying for the differencing service The snapshot facility must address four important issues Lastly, secuπty and pπvxy are lmpoπαr.t Becα.se tne use of CGI. synchronization, resource utilization, and secu- πty/pπvxy CGI scπpts run with minimal pπviieges. from an xcoar.t to which many people have xcess. the data in me repositon is CGI is a problem because there is no way for snapshot to vulnerable to any CGI scπpt and any user witn xcess to the interxt wtih the user and the user's browser, other tan by CGI area. Data in this repository can be browsed, altered, or sending HTML output When a CGI scπpt is ιnvok& hupd deleted. In order to use the fxihty one must give an identi¬ sets up a default timeout, and if the scπpt does not generate fier (currently one's email address, which anyone can spec¬ output for a full timeout interval, htφd will return an error ify) that is used subsequently to compare version numbers to the browser. This was a problem for snapshot because Browsing the repository can therefore indicate whicn user the scnpt might have to retπeve a page over the Internet and has an interest in which page, how often the user has saved then do a time-consuming companson against an archived a new checkpoint, and so on. version. The server does not tell snapshot what a reasonable By moving to an authenticated system on a secure ma¬ timeout interval might be for any subsequent retπevals; in¬ chine, one could break some of these connections and oo- stead this is hard-coded into the scπpt. In order to keep the scure individuals' xtiviues while providing oetter secuπty HTTP connection alive, snapshot forks a child process that The repository would associate impersonal xcount identi¬ generates one spxe charxter (ignored by the W 3 browser) fiers with a set of URLs and version numoers, and pass¬ every several seconds while the parent is retπeving a page words would be needed to xcess one of these accounts or executing htmldiff Whoever administers this fxihty. however, will still have Synchronization between simultaneous users of the fxil- f ormauon about which user xcesses which pages, unless ny is complicated by the use of multiple files for bookkeep¬ the xcount creation can be done anonymously ing The system must synchronize xcess to the RCS repos¬ itory, the locally cxhed copy of the HTML document, and the control files that record which version of exh page a user has seen. Currently this is done by using UNIX file 4 HtmlDiff: Comparison of HTML locking on both a per-URL lock file and the per-user con¬ pages trol file. Ideally the locks could be queued such that if mul¬ tiple users request the same page simultaneously, the sec¬ In our expeπence. only a small frxtion cf pages on the ond snapshot process would just wait for the page and then VV 3 contain information thai allows users to ascertain how return, rather than repeating the work. This is not so impor¬ the pages have changed — examples include icons that high¬ tant for making snapshots, in which case a proxy-cxhing light recent additions, a link to a "chan eiog". or a spe¬ server can respond to the second request quickly and RCS cial "What's New" page. As was mentioned in the intro¬ can easily determine that nothing has changed, but there is duction, these approxhes sutfer from deficiencies Tnev no reason to run htmldiff twice on the same data. are intended to be viewed bv ail users, but users wi' 1 \ ιsιt The latter point relates to the general issue of resource the pages ai different intervals and have different .ceαs oi utilization. Snapshot has the potential to use large amounts "what s nev." In addition, the maintamer must ot both processing and disk spxe. The need to execute generate the list of recent changes, usuailv bv mar.uall Trackmε and View ε Chanees on the Web (NO HANDS) marking up the HTML. subsequence (LCS) problem [4.5] (with several speed opti¬ Automatic compaπson of HTML pages and generauon mizations) to compare HTML documents. This is the well- of marked-up pages frees the HTML provider from hav¬ known compaπson algoπthm used by the Unix diff util¬ ing to determine what's new and creating new or modified ity [7]. The LCS problem is to find a (not necessaπly con¬ HTML pages to point to the differences. There are many tiguous) common subsequence of two sequences of tokens ways to compare documents and many ways to present the that has the longest length (or greatest weight). Tokens not results. This section descπbes vanous models for the com- in the LCS represent changes. In Unix diff. a token is a tex¬ panson of HTML documents, our compaπson algoπthm. tual line and exn line has weight equal to 1. In htmldiff. a and issues involved in presenting the results of the compar¬ token is eitner a sentence-breaking markup or a sentence, ison. which consists ot a sequence of words and non-sentence- breaking markups. Note that the definition of sentence is not recursive; sentences cannot contain sentences. A sim¬ 4.1 What's in a Diff? ple lexical analysis of an HTML document creates the token sequence and conveπs the case of the markup name and as¬ HTML separates content (raw text) from markups. While sociated aπabie.vaiue) pairs to upper-case: parsing is not many markups (such as <P>. <I>. and <HR>) simply required. change the tormatting and presentation of the raw text, cer¬ tain markups such as images (<IMG src= > and hy¬ We no descπbe how the weighted LCS algoπthm com¬ pertext reterences (< A href= >) are "content-defining " pares two tokens and computes a non-negauve weight re¬ Whuespxe in a document does not provide any content flecting tne degree to which they match (a weignt ot 0 denotes no match). Sentence-breaking markups can onh (except perhaps inside a <PRE>), and should not impxi companson. match sentence-breaking markups. They mus" be iden¬ tical (modulo whuespxe. case, and reordeπng 3 f (vaπ- At one extreme, one can view an i TML document able.value) pairs) in order to match (see section -..3 for a as merely a sequence of words and "content-defining" discussion of the ramifications of this). A match has weight markups. Markups that are not "content-defining" as well equal to 1. Sentences can match only sentences, but sen¬ as whuespxe are ignored for the purposes ot compan¬ tences need not be identical to match one another. We use son. The fxi that the text inside <P>...</P> is logically two steps to determine whether or not two sentences match. grouped together as a paragraph is lost. As a result, if one The first step uses sentence length as a companson metπc. took the text of a paragraph compπsed of four sentences and Sentence length is defined to be the number of words and turned it into a list (<UL>) of four sentences (exh staπ- "content-defining" markups such as <IMG> or <A> in a mg with <LI>). no difference would be flagged because sentence. Markups such as <B> or <I> are not counted me content matches exactly. If the lengths of two sentences are not "sufficiently close." At the other extreme, one can view HTML as a hier¬ then they do not match. Otherwise, the second step com¬ archical document and compare the parse tree or abstract putes the LCS of the two sentences (where words matching syntax tree representations of the documents, using sub¬ exxtly against words are assigned weight 1. and markups tree equality (or some weaker measure) as a basis for com¬ match exxtly against markups, as before). Let W be the paπson. In this case, a subtree representing a paragraph number of words and content-defining markups in the LCS (<P> </P>) mιght be incomparable with a subtree rep¬ of the two sentences and let L be the sum of the lengths resenting a list (<UL>...</UL>). The example of replx- of the two sentences. If the percentage (2 * W)/L ιs suf¬ ing a paragraph with a list would be flagged as both a con¬ ficiently large, then the sentences match with weight W tent and format change. Otherwise, they do not match. We view an HTML document as a sequence of sentences and "sentence-breaking" markups (such as <P>, <HR>, <LI>, or <H1>) where a "sentence" is a sequence of 4.2 Presentation of the differences words and certain (non-sentence-breaking) markups (such as <B> or <A>). A "sentence" contains ai most one En¬ The compaπson algoπthm outlined above yields a mapping glish sentence, but may be a fragment of an English sen¬ from the toicens of the old document to the toκens of the new tence. All markups are represented and are compared, re¬ document. Tokens that have a mapping are termed "com¬ gardless of whether or not those markups are "content- mon", tokens that are in the old (new) document but have defining." In the paragraph-to-Iist example, the companson no counterpart in the new (old) are "old" ("new") We refer would show no change to content, but a change to the for¬ to the "old" and "new" tokens as "differences" matting. We investigated three basic ways to present tne differ¬ We apply Hirshberg's solution to the longest common ences bv creating HTML documents that nieh aht the dit- Tracking and Vi wing Changes on the Web (NO HANDS) ferences with a vaπety of markup techniques: to areas in the document that have changed. A red arrow Side-bj -Side A side-by-side presentation of the docu¬ points to old content and a green arrow points to new con¬ ments with common text vertically synchronized is a tent. The arrows are also intemal hypeπext references to very popular and pleasing way to display the differ¬ one another, linked in a chain to allow quick traversal of ences between documents (see. for example. Unix sdiff the differences. A banner at the front of the document con¬ or SGI ' s graphical diff tool gdiff). Unfoπunateiy. there tains a link to the first difference. Old text is displayed in is no good mechanism in plxe with current HTML "stnick-out" font using <STRIKE>. which in our experi¬ and browser technology that allows such synchroniza¬ ence is rarely used in HTML found on the ll" 3 . Unfortu¬ tion (although it might be possible to make a document nately, there is no ideal font for showing "new" text. We that contained a table with a document per column in currently use <STRONG ><I>. Ideally, we would like to wnich rows of the table were used to xhieve synchro¬ be able to color code the text or text bxkground to high¬ nization). light old and new text, but this capability is not provided by current browsers. Another approxh would be to choose α Only Differences Show only differences (old and new) font that is not xtive at the point of the difference. ana eliminate the common part (as done in Unix diff). Note that not all changes in the documents are nigh- This optimizes for the "common" case, where there lighted. For example, new markups that are not "ccntent- is much in common between the documents. This is deπning" (such as <P>) are not marked up. However, especially useful for very large documents but can be markups such as anchors are highlighted. Consider :ne ex¬ contusing oecause of the loss of surrounding common ample of changing the URL in an anchor but not the content context. .Another problem with this approxh is that an suσounded by <A>...</A>. In this case, an arrow will HTML document compnsed of an interleaving of old po ' to the text of the anchor, but the text itself will be in and new fragments might be synthetically incorrect. its c ginal font, signifying a change to just the URL. Merged-page Create an HTML page that summaπzes all of the common, new. and old mateπal. This has the advantage that the common material is displayed just 4.3 Issues and Extensions once (unlike the side-by-side presentation). However. Since htmldiff can parse an HTML document and rectify incorporating two pages into one again raises the dan¬ certain syntxtic problems, such as mismaiched or missing ger of creating syntxtically or semantically incorrect markups, the only real problem it is likely to encounter is a HTML (consider convening a list of items into a table. set of changes that are so pervasive as to make the resulting for example). merged HTML unreadable. For instance, if every other line Our preference is to present the differences in the were changed, then the mixture of unrelated struc -out and merged-page format to provide context and use intemal hy¬ emphasized text would be muddled. We are expenment- pertext references to link the differences together in a chain mg with methods for varying the degree to which oid and so the user can quickly jump from difference to difference. new text can be interspersed, as well as thresholds to spec¬ We currently deal with the syntxtic/semanuc problem of ify when the changes are too numerous to display meaning¬ merging by eliminating all old markups from the merged fully. page (note that this doesn't mean all markups in the older Currently, htmldiff is neither "version-aware" nor " eb- document, just the ones classified as "old" by the com¬ aware". Thai is. htmldiff only compares the text of two paπson algonthm). As a result, old hypertext references HTML pages. It does not compare versions of the enti¬ and images do not appear in the merged page (of course. ties that the pages refer to. xcess them, or invoke itself re¬ since they were deleted they may not be xcessible any¬ cursively on other referenced pages. This has a number of way). However, by reversing the sense of "old" and "new" consequences. The good news is that htmldiff does not in¬ one can create a merged page with the old markups intact cur the overhead of pulling versions from a repository or and the new deleted. A more Drxonian option would be to sending requests over the W 3 for information. This cost is leave out all old mateπal. In this case, there are no syntx- consumed by w3newer and snapshot. The bad news is that :;c proDiems given that the most recent page is syntactically some differences may be ignored. For example, if the con¬ correct to begin with: the merged page is simply the most re¬ tents of an image file are changed but the URL of the fiie cent page plus some markups to point to the new material. does not. then the URL in the page will not be flagged as We are exploπng other ways to create a merged page. changed. To support such compaπson would require some .An example of htmldiff 's merged-page output appears in sort of versiomng of referenced entities and would also re¬ Figure 1. Markups are used to highlightold and new mate- quire hmύdiff to have access to the version repositoπes. r.al as follows. Two small anow images are used to point Full versiomng of all entities would allow interesting com- rxJc-.-gaπd Viewing Changes on the Web (NO HANDS) BLANK PAGE Tracking and Viewing Changes on the Web (NO HANDS) paπsons to be done, but would dramatically increase stor¬ 6 Extensions age requirements A cheaper alternative would be to store a checksum of exh entity and use the checksums to deter¬ This section descnoes some possible extensions to the work, mine if some " thιng has changed. We are explonng how to already presented. Section 6 1 discusses an tnterfxe be¬ efficiently periorm such "smarter" compaπsons. tween RCS and htmldiff \ a\ is already implemented, while Sections 6.2 and 6.3 presents unimplemented extensions to integrate tracking modifications into the server and to in¬ 5 Integrating the tools voke scnpts via the HTTP POST protocol. There are two entry points HANDS, one through 6.1 Server-side Version Control directly by the user, prob¬ The tools αescπbed above do not require any changes io ar¬ ably oy a cr entry, and generates an HTML docu¬ bitrary servers or clients on the W 3 Existing GET and POST ment indicating which pages have changed. If specified. protocols are used to communicate with speciήc servers that w3 e*er will associate three links with exh document in save versions ot documents and provide marked-uπ ver¬ the hothst: sions showing how they have cnanged. However, if a ser. er runs htmldiff and some perl scnpts. it can provide a αirect Remember Send the URL io the snapshot fxility. to save version-control interfxe and avoid the need to store copies a copy of the page. Though the page is retπeved. the of us HTML documents elsewhere. RCS ci command ensures that it is not saved if it is un¬ The perl scnpts we have wπtten provide an lntenxe to changed ' ' rom the previous time it was stored away. RCS [14] A CGI sr tpt (/cgi-bin/rlogj converts the out¬ put of rlog into HTV L. showing the user a history ot the Diff Have the snapshot fxi ty invoke htmldiff to display document with links tu view any specific version or to see the changes in a page since it was last saved away by the differences between two versions. Another scnpt (/cgi- the user bin/co) displays a version of a document under RCS con¬ trol, while still another (/cgi-bin/rcsdiff) displays the dit- History Have snapshot display a full log of versions of this ferences. If the file's name ends in html then htmldiff is page, with the ability to run htmldiff on any pair of ver¬ used to display the differences, ra ier than the rcsdiff pro¬ sions or to view a particular version directly. (See Fig¬ gram. ure 2.) As an example, one might set up a Last-Modifieα field at the bottom of an HTML document to be a linx to the r log Thus, exh page that is reported as "new" can immedi¬ scnpt, with the document name specified as a parameter ately be passed to htmldiff. and any page in the list can be After clicking on this unobtrusive field, the user would be "rememoered" for future use. An example of w3newer' able to see the history of the document output appears tn Figure 3. A user may also to choose to enter snapshot directly to check -in pages, or view the current page or the version his¬ 6.2 Server-side URL Tracking tory Figure 4 shows the lnterfxe to NO HANDS through Currently. w3newer runs on the user's machine, so multi¬ snapshot. If the user selects the history link, the page shown ple instantiations of the scπpt may perform the same work in Figure 2 is presented. Finally, selecting two pages to .Although it runs a related daemon on the same mxhine as compare invokes htmldiff, as in Fipre 1. an AT&T-wide proxy-cxhmg server, which returns infor¬ One disadvantage of the current approxh is that there is mation about pages that are cuπently cxhed on the server no direct interxuon between w3newer, snapshot, and the and may eliminate some xcesses over the Internet, there is IV 3 browser Viewing a page with htmldiff does not cause insufficient locality in that cxhe for it to eliminate a signif¬ the browser to record that the page has just been seen; in¬ icant frxtioπ of requests. stead, the browser records the URL that was used to invoke Alternatively, w3newer could be run on the set ot pages htmldiff in the first pixe. Subsequently. w3newer uses the that have oeen saved by the snapshot dx on Regardless obsolete datestamp from the browser and continues to re¬ of how many users have registered an interest in a page, port mat the page has been modified more recently than the it need only be checked once: if changed, the new version orowser has seen it. As a result, the user must view a page could be saved automatically Then a user could request a αirecuy as well as via htmldiff in order to both remove it list of all pages that have oeen saved awav. and get an indi¬ tra tne list ot modified pages and see the xtual differ¬ cation of which pages have cnanged since tney were saved ences by the user Tracking and Viewing Changes on the Web (NO HANDS) BLANK PAGΞ Tracking aπc Viewing Changes on the Web (NO HANDS) Adding this functionality would be useful, since it would to invoke. offer economies of scale. It would have the disadvantage Instead, the browser could be modified to e oetter sup¬ of being decoupled from a given user's W 3 browser his¬ port for forms: tory; i.e.. if a user views a page directly, the snapshot facil¬ ity would have no indication of this and might present the • It should store the filled-out version ci a toπr. ,n its page as having been modified. bookmark file, so the user could jump directly to the output of a CGI scnpt. 6.3 Interactions with CGI scripts • It should be able to pass a foπn directly to NO HAV DS, along with the URL specified in the FORM tag. so that Because NO HANDS can handle arbitrary URLs, it can inter¬ the output could be stored under RCS. act with CGI scnpts that use the GET protocol by passing ar¬ guments to the scnpt as part of the URL. However, services that use POST cannot be accessed, because the input to the 7 Conclusions services is not stored. Both w3newer and snapshot would have to be modified NO HANDS combmes notification, archiving, ana ditferenc- to support the POST protocol, in order to mvoke a service ing of W 3 pages to a single cohesive tool It x ieves and see if tne result has changed, and then to store away the economies of scale by avoiding unnecessary HIT? ac¬ result and display the changes if it has. The lnterfxe to NO cesses, saving pages at most once each time they are mod¬ H \N DS to support POST is unclear, however. A user could ified (regardless of the number of users wno track r. . and manually save the source to an HTML form and change using RCS as the underlying versiomng system. Automatic the URL the form invokes to be something provided by NO generation of differences within the HTML ιramewor< pro¬ HANDS. It. in m. would have to make a copy of us input vides users with the ability to see both insertions anc aeie- to pass along to me xtuai service. The result would be an uons in a convenient fashion. HTTP equivalent of a UNIX pipe, interposing an extra ser¬ In the general setting of the W 3 and document al. vice between the browser and the service the user is trying NO HANDS benefits two communities users ol ihe no Tracking and Viewing Changes on the Web (NO HANDS) longer have to browse to find pages of interest that have [7] J W. Huniand M. D. McDroy An algorithm lor duiereni.ii changed; HTML providers no longer have to create suit¬ file compaπson Technical Repoπ Computing Science TR ably marked-up pages to show "what's new". While such #41. Bell aboraiones. Murray Hdl. N J . 1975 automation is clearly helpful m this general context, we ex¬ [8] M. Nelson. B. Weich. and J. Ousierhout. Caching in ine pect that NO HANDS will be a cntical part of more focused Sprite network file system. ACM Transactions on uses of the V 3 , especially in areas involving collaborative Computer Systems.6(1).134-154. February 198S and distributed work. [9] M. Ne bery Kaupo. Several issues sdll need to be addressed. In particular, b.lp://wwwvuw.ac.nz/ " newberγ/Kaupoiιtn.l many of the complications of NO HANDS could be avoided [ 10] A standard for robot exclusion by better integration with W 3 browsers ana servers. For in¬ hup.//webjιexor.co.uk/ma /doc/roboιs/noroDou ln-;-.! stance, viewing the difference between an older version of a page and us current version should update the browser 's no¬ [11] David S. Rosenblum and Balachander Kjisnnaniurun Generalized eveni-acuon handling In Baiaciiand.r tion of wnen the page was last visited. Finally, the increas- jishnamuπby. editor. Practical Reusable UNIX S fr.mre m. .. ailabiluy of distπbuted, hierarchical HTTP reposito¬ chapter 9 John Wiley &. Sons. New York. 1995 ries such as Harvest [1] will be both an opportunity and a [12) R. Sandberg. D Goldberg, S. leimac. D Walsh. -.-.Ϊ challenge for scalanle notification mechanisms and version B. Lyon Design and implementauon of the Sun etwork archives filesysiem. In. Proceedings of the USEStX 1985 Summer Conference, pages 119-130. June 1985 [13] Sun Microsystems. The Hoϋava Browser White Pcp r Available as http://java.sun.eom l.0alpE-.3.doc/o\ er- view hotjava/browser-wbiu paper.ps [ 14] W. Tichy RCS a system f version control Softwar — Practice &. Experience. 15(7) 637-65-!. 1985 [15] Url-minder hu^://wwwjιeuιiffld.con URL-mιnder/URL-mιnderJιuιιl [16] WaπnlisL hup://glιmpse.cs.an zona.edu: 1994/"paul/waπulιsι References [1] C. Mic Bowman el al. Harvest: A scalable, customizable discovery and access system. Technical Report CU-CS-732-94, DepL of Computer Science, University of Colorado-Boulder. March 1995. [2] B. B. Cutter IE. w3new. hUp://www.snι£f.com bcuuer/pro- gτams w3new/w3newj.ιml. [3] James S. Gwenzman and Margo Seltzer. The case for geographical push -caching. In Proceedings of the Fifth Workshop Hot Topics tn Operating Systems (H0T0S-V), pages 51-55, Orcas Island, WA. May 1995. IEEE. [4] D. S. Hirscbberg. A lmear space algorithm for computing maximal common subsequences. Communications of the ACM. 18(6).341-343. June 1975. [5] D. S. Hirscbberg. Algorithms for the longest common subsequence problem. Journalofthe ACM. 24<4):664-675. October 1977. [6] J. Howard et al. Scale and performance in a distributed file system. ACM Transactions on Computer Systems. 6(1):51-81. February 1988. COMPUTER PROGRAM LISTING structure html dιt( - slruct r c 1 ind 00 open StrmgUtil end open Html fun markup si s2 11 12 - DIFPlif si = s2 and lso 11 - 12 datatype diff - SAKE | NEW | OLD | NEWERASE then 1 else 0.(1) fun w I (RAH si), (RAM «2|l - fun is . visible ISEQ(l) |BKEND(1)| = If i = s2 then DIPP(1,||| else DIPPIO.III (List.exists (fn x => is.visible x| 1) I v l(BEGEND(U)).IBEGENO(12H) = is.visible IRAN(s)) = true let is.visible (HAR UP(-IMG\J) = true val m = reallminlr lengthllll , is.visible (NAKKUPCV, 111 = r encjthlUIII List exists |fn(x,y|=> (x^HREP'll 1 val a = real («aι(r_length| 111 , I is.visible h = false r_length(12|ll in fun has.text ISEQIl)|BECEN0(1|| - if (ma = 0 0 orelse (mi/ma) < 0 61 then (List.exists Ifn x => has.text xl 1) DIPPIO.III has.text IRAW( ) = true else has.text _ = false let val DIPPIw.ll = (diff I' expression comparison *) (Vector vector ll) structure HtmlOrder : ORD.KBΪ = (Vector vector 121) struct val re = .common 11 I type ord . key = string; in fun cmpKey (sl,s2) = StringUtil strcmp(sl,s2) if (ai i ) 00 andalso lreal|rc|)/ai > 0 θl t IND end, -4 hen OlPPIrc. II ( • sets of sii inys 'I else structure hmap = SplayDiclllltαlOrdeil, DIPPIO.III end ( • diffing html docs '1 end structure HtmlWeight HEIGHT = v llml as IKλRKUPUl, 11111. struct (■2 as (MARKUP(s2,l2llll -- type T - HtmlDoc markup si s2 11 12 fun weight diff Ihl.h2l = w llml as lOMtt KUPISl, lllll, let |m2 as IEN0HΛJtiπjP(s2, 12| 11 I = fun r_length I = markup si s2 11 12 let w ((ml as IPRCαWMUP l, 1111), fun r_l (x-.xs) c = |m2 as (PR0CMAIlltUP(s2, 121) I) if is.vislble x then markup si s2 11 12 r_l xs (oil w llml as IDTCUWtiαJPIsl.lllll, else Im2 as (DecUURKUP(s2.12)111 r I xs c markup si s211 12 I r_l |T c = c w (_,_| = DIPPIO.III in r 1 0 hl,h2l end end fun r_comnon 1 ind = end let fun r t Ix xs) (li.j.i) ysl k c = structure HtmlUiff = Dif fFUNIstructure M = lltmlHeighil it i = k then if is_vιsιble x then (' functions for dealing with whitespace in RAH 'I r c xs ys (kill (ell fun elι-ιinate_whιte h = lse let r i xς y. (k«l( 1 fun is taw white (KΛWIsll else lei i <. «•- Hi. j.il ysl Ik.1) fun whitespace s I .. let 0 f fiθct23,1995| /home/tball/sml/mlweb/html_diff.sml . 7:18:12 λ king — 09:47 Aυg2 J® o v.lI l 1.1 II. I.'n n lun I l 11 Jl si val M - l.l 0 flll.fun esc.bool esc fun eικl Inn val "..ind = ref b.len let in fun iter i = IλlgB a i b b.len Ll false I. if i < πgthl'ai) then IλlgB la'il la.len-il b b.len L2 true), (update!!ar,i.fill_fun il, N • • Iwt b.lenl; if (esc.bool il then while Cj) >= 0 do (esc.fun i| I else if CH) <= Int Cj)) then iter lull) (N = (wt (!j)l: H.ind • = Cjl) else else lend.fun II) 111 j = ϋjl - I ■ter st I; end I (AlgC (a<ιl (a.len-i) fun simple.fill ar st fill.fun = (bt('M.ind)l Ib.len-l'H.indll fill ar st fill.fun (Aloe a i b IIN.indl resultl) Ifn x => false) Ifn > => I)) Ifn x => II) I end fun comp.weight IrO.rl) IvO.vl) j I = let fun cmb || w I = DIFPI*. II val DIPPIw,.) - I cmb (|w,d| >s) sum I = cmb xs (wsuml Id II (weight (Vector sub(!v0, i 11, (cmb (AlgC I 1 n HI 0 HI Vector subl'vl, j 1)11 maxlsubl'iO.i I «. lun dilf IΛ W T vfClot) IB W T vector! = maxlsubCrl.i-H.subl'rO.ill) Ip cliff 0 0 A Bl end I' Wuu Yang's optimized 1£S algorithm 'I fun verify.column (col,rowl (x.yl fun lies (A N T vcctoil IB I T vector I if col <- x then lot fill Coll row open Array lcomp weight val weight = (W weight dil(I (ColO.Colll Iref B, ref λ| coll val m = Vector length A Ifn r :> IsubllColl.rl > subl'RowO.xll I val π = Vector length B Ifn r => ((simple.fill Coll (cl) val RowO ref (arrayI(m«II.0)I (fn s --> (subϋColl.rlll). val Rowl = tef (aιιay((m>l| ,01) (swaρ.ref(ColO.Coll)l; val ColO = tef (arιay((n'l| ,0)1 Iverify.row (col«1.y•11 (col.rllll val Coll = ref larraylIn.1),011 Ifn . => (swap.ref(ColO.Colll, I* afi'.timcs that RnwO ' row J 1 and CπlO - rol I 1 verify.column (col«l,ιowl (x.yl)l ' have already been computed ') else lun findjuvot 1 J = (x.yl let (un swap ref (x yl and verify tow (col. row) (x,y) : let val t = x in if row < ■ y then (x -I'yl.y =l't)l fill Rowl col end (comp.weight IR.iwO.Rowll lid A I. I HI ι..w) signatu sig " 11 4=» OO end structur I (OWIF-NTlsl > noim IKAHIil acil end j ( ■ ■ < • 'I (IPKN -> norm Imarkiipll ■ ace) j c • </ ■ •IKNIXH'KN • > fun parse s - let let val u markup!) val _ -- HtmlLex UseiDeclarations filename =s val cl = closeofI mu ) val . - HtmlLex.yylineno:= 0 in val lex = HtmlLex makeLexer loffer si norm (cl accl end parse' lex I (' • <• • 'I DECLOPEN => end let val mu - markup11 fun parse . strea (in f. fl - val cl = declof( mu I let in val _ = Ht lLex UserDeclarations (il norm lcl accl val . = HtmlLex.yylineno:* 0 end val lex * HtmlLex. akeLexer (inputc lin.fll I (• • <' • •) PROCOPEN => val _ - LexHelp.SetFilename ( let val mu - markup!) parse' lex val cl - procof! mu I end in norm lei accl fun parsef f = end let val in.f = (open.in fl C l O I < ;> norm accl val lex = HtmlLex.HkeLexer Iinputc (in.f)) val . = HtmlLex yylineno:= 0 and att11st first - val _ = LexHelp SetFilename f lease first of val res - parse' lex IDIs) => lease next . tok11 of EQ -> lease next . tok!I of Kclose.in in.fI; resl IDIrl -> Itoupper s.rl end I STR(r) => Itoupper s.rl => C'BAD'V'BAO' I lattlistlnext.lokl • *())) fun doc ap f doc = CLOSE => II (case doc of IDIr) => Itoupper s. "I lattl ist l lDlrl l l SEQ(l) => SEQ ( (docmap f) 1 ) STfUrl => Itoupper s,"l lattlist ISTR(r l l ) I x => f x) => raise BadHtml) STRIsl => K'V " s - -\" l , "l (attl isl (next.tokl l l l fun getatt aname (|n,v) .tl = x --> I I I if n = (jcrlib toupper anamel then SOME v else getatt aname t and eat .conxneπt s - I getatt aname 11 = NONE (case next.tok! I of CUKECOMHBrT => (s " ■ - ' I fun info.stream linS, si ~ I CUMM NTImorel => eat comment Is ' morel let j . = > raise Badlltml ! val html = parse stream linS. si fun fields IHARKUPls, attlistll - and markup I) lease s of lease next tok|| of • TEX.TAREA* > I getatt 'NAM-:' altlist lDlsl => HARKUPIt.mlXjPi s.attlistl nexl lukll )) I 'SELECT- -> I getalt 'NANK' -tllisl I I 0PENCf)r«liKT => MARKUPI • - '.Heat lommcπl ■■ ,")! I 'INPUT' • > I getatl 'NAME' all list I X => MARKUP! •• .nilll I => III j fields ISEOII'lals)) = datatype iexresult = OPEN I ΞNDOPΞN | DECLOPEN | PROCOPEN | CLOSE | OPENCOMMΞNT \ CLOSECOMMENT | EQ | CONTENT of string COMMENT of string STR of string | ID of string | EOF vai eof = fn ι => EOF val filename ref " ■ ir.t (s (makestring (0) \n")) \1 error = fn (x.l.int) => outpu (std_err,x line (makestring (1+1) ) A file ( !filename) ' \r." ) %% %s MU COMM COMMSTUP; %structure HtmlLex ws = ([vt\ \n])-; nonws = { 'y \ \t\n<>=] )+; %count %% <INIT:AL>[ Λ <>] * => (dprint "CONTENT" : CONTENT(yytext) ) ; <:NIΓ:AL>"<" => (YYEΞGIN MU; dpnr.t OPEN"; OPEN) ; <INIT:AL>"<' => {YY3ΞGIN MU dprint "ΞNDOPEN"; ENDOPEN) ; <INIT:AL>"<: => (YY2ΞGIN MU dprint "DECLOPEN" ; DECLOPEN) ; <INIT:AL>"<? => (YYBΞGIN MU dprint "PROCOPEN" ; PROCOPEN) ; <MU>">" => (YYBΞGIN INITIAL; dprint "CLOSE"; CLOSE); <MU>"--" => (YY3EGIN COMM; dprint "OPΞNCOMMENT" ; OPENCOMMENT) ; <MU>"=" => (dprint "EQ" ; EQ) ; <MU>\" [. A \n"] "\" => (dprint "STR"; STR)substring(yytext, 1, String.size(yytext) -2 ] <MU>{ onws) => (dprint "ID"; ID(yytext)) ; <MU>{ws) => (lexO ) ; <COMM>"--" => (YYBEGIN MU; dprint "CLOΞΞCOMMENT" ; CLOSECOMMENT); <COMM>"<" => (YY3EGIN INITIAL; dprint "CLOSECOMMENT"; (yybufpos := ( lyybufpos)-1) ; CLOSECOMMENT); <COMM> ['-<>]* => (dprint "COMMENTTΞX " ; COMMEN (yytext) ) ; => (error("html lex: ignoring bad character " A yytext, lyylineno) ; lex() %% I ' /usr/locai/cin.'per -- I A cement mentioning per., to prevent looping aval 'exec perl -5 Si S { 1-'S»' ' if 0 ; • For the laces, info on Fred Dcuglis ' s update ci vl . O of wjnewer. sae • I htt . / /wuw.l.S . research .act . com -dougl s/ track.-ris/ t • For the latest ir.ioraacion cr. 3rooks C.ccer' s version cf w3new, check the URL • • nt p: / .'www eπg.paraαVnβ.com/c;ι-bιn/ibcurn?user * Dcucterι.pkg * w3nβw • wjnewer l : - Creates a What s New list of πrcp. URL's ■ Kodifieα oy feed -ouglis <dcugiιsiresearcn.act. r=:α> ■ * .: separate user interface :rcm polling • w j newer is a program chat w ll excract a iisc cf ' RL' s from either your mosaic • .-.cclisc. αr will extract the URL' S free, a HTML ."ument It will then obtain • t.-.e HTTP ' l . l τ.ccιf icaticn caces tor ear.-, docucie-t listed, and output a HTML • file with tne URL s sorted cy tneir last modification time ?acκage . wjnewer Author . Sreexs C-ttεr '.bcucteriparadyr.e. com} Author Fred -oυς. s <-cu5l:s«researc.-. att. cj.> Latest version. l.C Last updated. Marc.-.. 1995 Archive w3newer.car.cz iIncludes everything you need tc ra it. except perl vl.0361 'Jhat t does. 1. w3newer first extracts the URL's from your hotlist or a HTML docunenc, and cached modification dates and access times from its own cached history and the WWW browser's history file. I . w3newer then .lands off the URLs to a separate process, which proceeds to do a HTTP/1.0 HEAD on each hctp. URL that sight be out of date and stores tne Last modified tme of each URL lif availableI 1. Finally. w3newer sorts its output basad on the last modified time of the document and categorises then by the month chey were modified. For non-htcp URL's or UXL's for which it can't retrieve the Last modified tme, it will l.st these at the bottom of ts HTKL output. This program was written because Broolts found himself frequently checking wee pages m ms hotlist to see f they had recently been cnanged. Now ne runs w3newer from cron nightly, and checks its output each morning. pointing w3newer at a list of ORL's ■ 1. Using your Mosaic hotlist by default. w3newer will read your Mosaic • hotlist file and excract the document URL's and document titles. It will • look for your hotlist in your hone directory as the file • -' .mosaic-hoclisc-default this can be ovar den by using the environment variable • W3NΣWΪR_H_TLIST • The default and environment variable can be ovemden w th the a contend l ne arguments -hotlist.fn or -i with a argument of the file in • Mosaic-notlisc format. i 2. Extracting URL s from a H M document f you call w3newer with the i -u argument and a HTKL document URL t w ll retrieve that document. ■ extract all the nxs in tne document, and c.-.ecx the modification tne of • each i rjc. • 3 Extracting URL's from a HTKL document and mime modification times ■ f you call w3newer with cne -html argument and a HTML document ' URL. t will extract the document linxs and recrieve the modification • times, but unlike 12 ι-u argumen J. t will reolace the modification ' times ithi tne document ratner than producing a sorted list. i jihen wjnewer parses the document, t w ll loot for a tag of the farm ■ <w3newer rl»'_?_,'> ' For example i <wιnewer riϊ'http-/. ww.nest.dom/f ie.htιnl'> « .-.he .-. : ' nss a :aς like t.-.e one aoove. .: ..11 recrieve che last modificaticn tune for the document specified by the ur * line, and include the last modification of that URL iif it existsI • For more information on the usage, run w3newer with the -' argument I • The latest information on this program can be found on the web - see above. » • This prog am was built with software written by others. • Their contributions are greatly appreciated.. t • libw -perl vθ.40 or later by Roy Fielding ifieldingiics.uci.edu) < http .' www. ics.uci.edu/KebSoft/l bwww-perl/ • βvaiuate_parameters levapi by Stephen 0. Lidie HusoliLehigh.EDU I I 'All the implementations of evaluace_parameters are available via • anonymous FTP from ftp.Lehigh.EOU 1128.1.0..] .41. Look in the ■ puo/evap/evap-2.x directory for the latest compressed tar file.' • perl 74.036 by Larry Wall :lwaπ»netlaba.coaι > ftp able fr-s ftp.uu.ne: :n /systeαs/gnu as peri-4.036.tar.gz • wjnew 'rt 1 by Brooks Cutter locutterSparadyne.cooi Known Bugs o none 'that I π aware of. Future work • 3 none (probably). I started to write some code that wou do a checksum • on a document specified by a UM. ... this would allow y. to track the • modification time basad on when the content cnanged... however I • never finished coding/ testing the checksum piece, and abandoned it. i you'll find the code commented out below.. • - picked up 1/95 by F. Doug s Cached modification unes are positive; i cached checksums are negative. • o eventually t hope co convert this into a www applet and let the ■ browser do the work, ϊou'ii find more info on www Applets at • http://www.let.rug.nl/-bert/W3A/W3A.ht-a • • c Other future work (FDI : multi-user 'database' of info. Types otber than i htt . t • Brooks Cutter (bcutcertparadynt.coml • http./ /www. eng.paraoyne.com/cgι-oιn/bbcurn?user'bcutcer $ ... . — i j newer is distrisuted under the Artistic License I included with your Perl i distribution files and with tne distribution of the libww-perl package) . • To run this program either set the variable below w th the location • S_jW(",43NEWE]t') < '/hone/tJball/src/w3nawer' unless ISEjW('WrølKlV)); SENV( 3NIWW ) < /h βe/tball; jrc/w3newer ' ; Sw3never_dιr • SEJWrwJNWBS'); unless I Sw3newer_dιr) I print «EOF: " • ERROR"* Unable to start wlnewer Sorry. 1 don't know wnere w3newer support files are installed 1 rlease either set the variable ■ S -3newer_dιr near the top of the w3newer script, :: set the W3NEHER environment variable to the oasc package directory exit: I iprocess.urls > 'perl Sw3newer_e:r. Iib/w3newer-get.pl". --..ess lSENVChttp_proxy') I ( SENV! no_proxy' ι «'wwwll26. research. att. om. zoo. att. com. radish.research.att.com. localhcst, tbu. att. ccs, ho. att. com' SENVt 'h tr._proxy )■ 'http://wwc.ih. act. corn:.000/'. • .-.hat version are you using? (set this if using Sw3r.ewer.dir: : 1 ιo_-vw_ver * '0.40' IZ :< 'L.S Wc'.PERL' i . • Sw3newer- r.'l:b lιbwww-perl-Slιo_-v-'_ver' , :.' defined SDJVI LIBWWW.PEF.L ; '.•'Inever ϊ:r' •Sw3newer_dιr/lib'. i whereis libwww-perl? I Either in the directory specified by LIBWWW.PKL • or under lit in Sw3newer_d r SENVI -LIBWW .PEJ -l. •/home/douglis/lib/perl'. ). Swnats.new - 'perl SENVl ' LIBWww.PERL ' l/POST http. '/notwβb. cou.att.com.'-dougiis/cgi-bin/whatsnew.cgi' • architecture-specific library Ifor sys/socket.ph: require 'arch.pl'; Sarch « iarch; if l-d ISlibloc » 'Sw3newer_dir/l b/arch/Sarch' ( unshiftlβEJC. Sliblocl: I nSdate 1 .-.' • TBALL re i re ' ccime.pl • •require lrequest.pl' : • part of www now require stat.pl' . require -www.pl' require 'wwwuri.pl'. require 'wwwerror.pl'. require ' wwdates . pi ' ; require wwwoot.pl'. I part of libwww-perl vθ.20.- -1 later.. unehiftlBDK.'/appl/avap/lib'l if l-d '/appl/evap'l . require evap.pl : require • parse.htal . i ■ ; t You shouldn't need to change anything below here. Sgl.requβsc.cimβouc > 45: 'months • I • January . 'February . 'March' . -April'. 'May, 'June'. 'July , 'August' . 'Septemoer' , 'October' , 'November' . 'December' •days* ( 'Sunday . 'Monday . "Tuesday , 'Wednesday . "Thursday . 'Friday . Saturday i . 5w3newer.author.home * 'stuff .com' ; Sw3newer_ver * '1.0': $w3rwwer_author.emaιl ■ 'bcutterβSw3newer_autJιor.hooe': Sw3newer.au.hor • <<EOF; <a href **hctp:/ /www. $w3πewer_auchor_bι»e/cgι-bin/bbcurn?user*bcucter*> Brooks Cutcer</a> EOF Sw3newer.oldurl * • bctp:/ /www. stuff . com/ -bcΛiccer/hcebm/progτanuι/w3rιewyw3nβw. tml' : Sw3newer_url ■ 'http://wwwll26.research.att.com/-douglis/traclc.u is/': Sw3newer_author2 ■ «EOF; <A HIlEF«'http: //www. research.act.com/orgs/ssr/people/douglis/ '>Pred Douglιs</A EOF I if URL isn't fully qualified, this is used wich .wwwurl' absolute 11 Sbase ■ fιle://loea.hosc. ISENVI'PWD'J II SENVCewd'll ''. SUserAgenc ■ 'wjnewer/ Sw3newer_ver Swww'Library' I Set up User-Age.-.;: header • Set the default '.'ser-Agent -wwwset.def.headeri 'http' . 'User-Agent ' .SUserAent) : «' thotl st.netscape = I "SENVl 'HOME' )/ .necscape-booicaarks.hCBl'. •5E3JVI 'HOffi' )/ .MCOM-boo«uar«..1t3 ') ; βhiscory.netscape = i 'SENVl 'HOME' 1/. netscape-history. ■ SENVl 'KOHE'l/.MCOM-globai-hstory'l : Shotlist nosaic = 'SEWV1 'HOHE' )/ .mosaic-hoclisc-defaulC. .hist.ry.-osaic = 'SENVl 'HOME' 1/ .mosaic-global-h story' ; Sd β faulc.browser « netscape': » This requires tne evap.pl library Sevap_pd:.<<EOF: p-r ■ 7? w.Rtwer 3D • the following options control output processing verbose v swicc cucpuc.fn 0 file ■ WjNEWEH fTKL '-/publιc.hcβi/w3newer_oucpuc html' ' If you prefer co nave it output to STDOUT uncoαnent t.-e next l ne ■ oucput.fr. 0 file • W3tiWra_HTHL. scdout 1 the following control where co get DHLs mosaic as switch netscape ns switch hociist.fn 1 f.ie « W3HEWER.H0T IST " history.fn h file • K3NTWER.HISTORY " url u string htni.uri heel scring • chese csneroi processing fro β .aonrsss from scring « DfAIL.AODRESS " checitsum ca oooiean • TROT » these control tne format of Che outpuc dispia/.neader ih boolean ■ TRUE dispiay.ϊcocer df boolean « TROT -lspiay.un nown dU boolean » TRUE iispiay.recenc Ir boolean » TRUE dl *Pla J _.r.cnanged du boolean ■ TRUE emboiden b swicch lasc. .sued .1 switch tiBest-πp.source ts switch • these control cacned information cache .-edt-iss := boolean • TRUE -a.fi.e ezif f..e » W3HEWΪR.CWILE • -/ w3newer.iiodc.aes' ca.def. threshold αadt string » M3liTWj».CT.DEF.THlEi) 'li' ant.iatest .1 boolean ■ FALSE ca.cbsolete.threshold emot scring - wSNEHER.cy.CES.TKJ'SH '3d' cacne.t.ιresholds.fn cfn file » W3NF " CTFILE • - - r.ewer. thresholds ' • miscellaneous options, esp for debuggi ng debug . integer • 0 max.urls siax.urls integer » -1 multi.user mu switch • nop n switch • false 1 integrated into mcdcimes • w3newer_cs_fιle csf file » WNEWER.CSriLE ' -< w3newer_=heαuuns ■ snapshot s swicch snapshoc.uri su scring » W3HEWER_SMAi»_URL "hcCF / 'radisn reseercn ace ccβ/cgi-biπ/πo.hands' ignore. ooots ign swicch fαras forms swicch PDTENB opcional f le lisc EOF *evap_pdc ■ splιcι , \n/ Sevap_pdc) Sevap.m ««E0F w j newer vswjnewer.vβr w3newer is a program that will excract a list of URL s from a hotlist ( either Netscape or Mosaic formic I , or will extract the URL s from an HTML document :t checks the modification dates of URLs thac have not been cnecked recently and produces a reporc of URLs chac have been modified since cney were viewed Ic cries not to poll things coo often It uses the following techniques - If your browser s history shows something nas been accessed recendy ic isn c checked - If «newer checked something recently ic isn t checked - If w j newer knows something has changed since it was last visited --. marks it as new but doesn' t check for a more current date unless it deems the modification dace informacion to be 'obsolete' isee documencacion for ca.obsolece_chreshold) - The definition of 'recent' can vary from URL co URL based on speci icacicns in a 'thresholds' file Curre-:., w3newer only handles http URLs not ftp f.ie or others T'lis -a> cnange Exajspies w3newer w3newer -usage.heip wjnewer -' winewer - f.U.neip wjnewer -ns -0 -iv -v (netscape bold last visited verbose) /erbose "hen /ercose is set w3newer will print out the name of each URL • t ciecu and generate some other stacus mfoπnacicn :'.ec«_- -e * :-ecκsum .s set wjnewer will use checicsuna :c te.. .i somβcπing -as zee- zci.'..ez .'. the modif. cacion dace is _-avιι.»ε.e * Oct 24, 1995 Y /home/tbal l/src/w3newer/ 3newer .oucpuc.fn This specifies wnere che HTML output is placed (scdouc indicates standard output: . .display.header determines wnccner or noc co princ a header on che HTKL output display.footer decersines whether or noc co princ a footer on che HTKL output .display.ur nowπ deterair.es whether or noc co display hos s for which ic cannot find HTTP.'1.0 Last modification daces .display.recent decerair.es whecher or noc co display ORLs chac were recently accessed and r.c: checked during chis run. .dispiey.unchancjed decerair.es wnetner or noc to display URLs chac nave noc changed since cney were lasc seen. hocl sc.fr. The filename of your Netscape/Mosaic hotlist By default ic is sec co che file - • .necscape-icoianarks.ht l or -/ .mosaic-hoclist-default, depending on whecher che 'netscape' or 'mosaic option is sec . The defau.c can be overridden wich che environment variable W3NEWHI_H0T1IS . hisccry.fn The filename of your Netscape/Mosaic history By default ic is sec to the file -/ necscape-global-history or -/.mosaic-glcbai-nistory, depe-. r.; en -nether che 'necscape' or 'mosaic opcion is sec. The defa.lc can De overridden with che environment variable 3HEWEIt_KISTORY url wlπe.e: .ill recrieve the HTKL file specified by this argument, excract all hyperlinks and images from the document, and proceed co cr.ecic the lasc aodificetion cute for each URL. html_.ri if you call wjnewer with che - col aπjυmene and a HTML document URL. ic will excract the doeus- .t links and recrieve the modificacion cimes. but unlike the previous opcion l-u argument), ic will rep.ace che modificacion times wichin che document rather than producing a sorted list. When vjnever parses the document, it will look for a tag cf the fom <w3newer url''URL'> For exaiaple <w3newer url''http: //www. ose. oα'file.ht l'> When it finds a cag like the one above, it will retrieve che lest modification tune for che document specified by che url» line, and include che lasc modificacion of hat URL lif ic exists fro .address By d faulc. cne program will sec che HTTP/1.0 From: header to your Usemaee i specified by USER erivironraent variable) and your hostnane . If your hostname isn' t fully qualified l ie: doesn' t have a ■ • in ic) then ic will run ' dcjvainname ' and tack on che resulc lif any) . This value is also used for the snapshot arguments Isee below 1 . This value can be overrides by setting the environment variable MML UDRESS cr passing your email address with the -from flag .mulci.user This flag is noc yec implemented. It refers co sharing modification dates among multiple users, netscape Th s flag indicates that Netscape bookmark and history files should be used mosaic This flag indicates chac Mosaic hotlist and history files should be used .embol ir. This flag indicates chac certain items thac are deemed co be of speciil mcerest should be generated in boldface Currently this refers to the dates of items that have been modified since c ey were lasc seen .lasc.visiced This flag indicates chac the tune when a URL was lasc visited. acccri..-.; co :.-. history file, should be included in wjnewer output. .timβsc-oo.source This flag indicates that the source of information for a URL Iprcr.'-cacnin; server, polling, previous run. etc. ) should be r.c.o... in jne.er output This ::: :r. specifies che file chac caches modification dates of V?._s T.-.e cefa..c can be overridden by cne environment variable ■ nvstαto-zαi .. . cache nodcimes This option indicates that modification daces should be cached. If TiZil. cm.fiiβ is ignored. .cx.ief.threshold This option specifies the default frequency for checking. URLs, m the forn Ma. Hh. Nd. Nw for N minutes, hours, days, or weeits. respectively .wa.it. latest :: this swicch is sec. always gee che current modificacion dace of a ■ „?-. chac has noc been accessed recently, rather than snipping it f ts Known modification date is more recent Chan che time it was visited by the srowser and noc beyond the dace specified by che -caoc swicch. ~_:bsolece_chrethold This swiccn specifies when a cached modificacion dace should noc be ccnsidered t; se current even if ic indicates chac a URL has noc eer. seen since ic was modified. cac.-e.chresholds.fn T.-. s opcisn specifies che file that specifies now often different '-".li should be checked. If ic does noc exisc. Chen all URLs are c.-.ecxed wicr, a frequency determined by cne -cade opcicn. The format :: cms file is: URL <whιcespace> cime.specif cation Default ciae.spec f cation Obsolete ciae.specification -efaulc overrides che value of -emd . Obsolete overrides -cmoc. If tira.specification is 'never' the URL is skipped even if it is found :τ. the hociisc. The URL can be a paccern co macch aga nsc iperi regexp syntax) .dtr-g j .nβn debug is > 0 ditional debugging outpuc is generated. The higher :.-.« nuαcer. the more output there is. .-uue.urls Tins opcicn specifies a limit on the number of ORLs to process. primarily fcr debugging purposes. . napshot If sec. che output will contain links to che snapshot CSI daemon. . snapshoc.url 'JRi co use for snapshot. . icnore.nobocs Sec in order co bypass saved information abouc which URLs disallow robocs (all or ncne I . . cras Sec co produce a form for eech URL rather than direct hyperlinks. EOF ■evap.tm > splitl/vn/.Sevap.o) : die "Error recumed from evap\n' if u eva l*evap_pdc. 'evap_n»ιi '» l) ; • I If om. address was set. set che defaulc header., if I Sopcions ( ' f rom.address ' 11 I •www sec.def header! 'http', 'From'.SoptionsCfroa.a dress')); •' ) else ( locallSuser: • SENVl 'USER' ) || gedogin || (gecpwuidlS<))[0) tl 'Intruder!", I locallShostl > SENVl'HOST'l || choplSfoo » '/usr/ucb/hostnane' , || 'Unknown.hosc'. locallShosc) • Shottname'rQON: Sopcions C from address ' ) • join! 'I'. Suser, Shosuia e ' FQtHl ; 1 Sopcion.scring » ■ •; fcr "Diff. 'Remember'. 'Hiitory') ( Sopcion.scring « '<opcιon> S_ ■ ; 1 if ( I Sopcions r oucpuc.fn ■ ) eq ■ >-') || 1 Sopcions I ' output. n ' ) »- /"\»*\|/l) I openlOUT. Sopcions routput.fn' ) I II die 'Can'c wrice o Sopcions I ' output.fn ') : Si\n': i else 1 .ocallSoldmaski > uaωk: :f I -e Sopcions! ■ oucpuc.fn' 11 ( localliaryi ■ scacl.) : local iSmode) > Sar 1 SSTJrODE] , umasklSoldaasic I (-.mode .07771): if i-s Sopcion I 'output.fn') I ( rename I Soptions ( Output.fn '! , 'SoptionsCoucpuc.fn')-'! || print STDERR 'Warning: can't rename Sopcions I ΌUCCUC fn')\n'. ) cpeictCUT. ">Sopcιons( 'output.fn' ) ' I il die 'Can't wrice co Sopcions! 'outpuc. fn' ) : S!\n', — sk S.ldmask; Ξ:c-.ι:r.s( verscse ) * 1 if Sopcions! 'debug' I , I debug implies veroose. :f i' IScpcio s! 'mosaic ) || Sopcions! netscape') || Sapcionsl 'hct.ist.fn') ]| Sopcions! 'url'l || Sopcions! 'hcmi.'.rl'))) 5cpcions(Sdef...:.browser) - 'TRUE'; sub ncsuchurl : local iSurll • i_. . '. SopcionsI .tsug ■ i > 0) ( pnncf STT -J. 'Goc a URL iSurll we didn'c ask fcr We know abouc: local ISu: . for Su is::: <eys lurlsl ! sue fιr.i_fιrsc.e>:i:-.i.-.; ( ic s, .enanes * *_, lccaiiSfiiei fcreach Sfile inanes) I if i-e Sfile return -file. princfi'Ca.-. : i:τ.z an exist;-; file jci.-.i '.r. ÷nanesi if Scpciinsrdeoug' ). rec.rn • ■ . I if IScpcions 1 aosaic 11 ' Scpcions. iociist.fr.') = .fir.d.fιrsc.exιscιnglthdclιsc_mosaιcι. Sopc ons('hιsc:ry_fr.' I « .fir.d.firsc.exiscngl»hiscory.mosa c: : ) elsif : Sept lcr.s : ' - e tscape ' 11 I Sopcions! 'hcciisc.fn' 1 » _fir.d_firsc_ejc scingltnoclisc_necscapel; Sopcions! 'hisccr/.fn' 1 « -find.f.rsc.exisc ngleniscor/.necscapei; I l.load.checksua if Sopcions r checksum'! I : s read in cacned scdificacion daces . checksums, if any. Then if we • see URLs wich cached info, updace che cached info based on any new l info. e.g. 'lasc seen' daces. if IScpcions! 'cac.ie.modcimes' I) ( •cached.modcimes ■ bread cached modtimeslSopt onsI 'cm.file' )) . ) else I icached.modtunes • 11 : 1 if ISoptionsCh stcry.fn') ne ") 1 tvisic.cimes » .load history i Sopcions ('history.cn'll: ) if ISoptionsl'url'll I Surl > .wwwurl'ar.solucelSbase,Sopc ons('url'l); if I'.wwwDocaUswedlSuxl.SUserλgenc)) (•' die 'Sorry, buc the remote sice does noc allow robocs to retrieve that URL n": 1 surls ..extracc.iinks.desclSurl!; ) elsif ( Soptiansl'Scml.urπi I S'.ri ■ .wwwur. acsolucelSbase.Sopcionsl'hcαl.ur i):!' if l'twwwooc , a.---ad!Surl.SUserΛgent]) It' die "Sorry. r._: tne remote sice does noc allow robocs co reenr/e chac ' RL n' ) princ :iT idιsplay_.-.c3llSurll , exic: ) else ( I use 'hoclist." names co have qualified pattern matching on call by ref - i is chis necessary' local (ihodist.-r Is, ihcclisc.daces); -load.hcclisciSopticnsl 'hoclisc.f.r I . 'hotlisc.urls. 'hocliscdatesl : *urls » Xhotlist.uris , I lurls.accessed » t.-.cclisc.dates: I for Sir: liceys Suns' I :'. 'Surls.accessedlSurll ivisit.timesfSurll) [ cπnc STZZ-l. "*'amιns. '.'isic cure from hodisc for Surl newer than global hιs::ry '.-.' if Sopt cns! 'debug') .. Svisit.tunβslS Svisic.c-raϋS.rl) ■ Suπs.accessedlSurll . SstasrclS-rl' = 'hotlist' ile.-.ϊt.n «.βys ivisit.tisesi •» 01 { delete Sections: ersoldβ-r ) : sue t.-.resnold ; lccai (Surl l « 8. rβt-rr. S threshold I S.rl) ;J defιr.edlSthreshold(Surl) ) : -ocailSu). icreacn Su •ft.hrβsncldst i Su. Surl if Soptionsl 'debug' } > 0. rtv.rn S-pticr.s' '".ie:. threshold' 1. • r. tia.i∑a ur. ts ir. terns zt seconds) ."a. li'jr.lts * r όC. h 60. 'd'. .4 ...allScurrant. ■ :. -ctal (Sthis--. t St.-.ismuit) . ■ -.-.l.e 'St nits > ' . iSthis-r.it. St isx-it. iunits) • tunits: S-nits-' S:.-.ιs..-. :, « Sthαsmulc * Scurrent: Scurre.-.: * S_-. ts!Sth:sunιtl , sub str.sees ! io:ai iSoπj.tnres.-.i • »_. .ca. !$B.Snua.Sur.it.Stl » 0. 3: « Soπς.thrβsh -nile ISt ne ") ' if (St =- /never/) ι return -I. ! elsif 1st «- rnd-lUmhdwIK.'l/l I Snum > SI. Sun t ■ S2. St » S3. S« * ■ snua * Sun t I'MSunit'l: I elsif !St ■- /"\d»S/l 1 S •• St: return Ssi. I else { die 'Car. t parse threshold Sons.chresh' sub secs2str { local ISsl « •_: local ISu.Sres.Sn) : f iSs < 01 t warn 'Negative t me (Ss) passed to routine * . return " Sres * ". for Su I'w, d'. 'h', 'a' I ( die 'Units not initialised' if IdefinedlSunitslSul); while (Ss >• SunitslSul) I Sn--- Ss -= Sur.itsiSu: Sres • S.-.S ' if Sh: Sres = 'Slsls' if Ss. Sres = '--' :f Sres eq Sres. -e Scptionsl cache.thresholds.fn')I ( .:callsset_defa_.t.Sset.cDsoletβ> • 0: c;β h :THS Ϊ 3H. Scpticnsl 'cache.thresholds.fn' 1) II die 'Can't open Soptionsl'cache.thresholds.fn'l" pr r.::iS7t.?- : 'Thresr.elds.T.'l f Soptions ( ■ detaic ■ ) > 2: ScptionsCca.def .threshold') - iscrisec lSl) . Sset.default • 1 pri.-.tf(STDEER '\tt-30s »d\π' 'Default', Soptionsl 'ca.def. hreshold' I ) if Sopticnsl 'debug') > 2 ) elsif '/"Obsolete\s«t 'l/il 1 Scpticasl ca.obsolete. hreshold ) » -str2secs(Sl) . Sset.cosclete ■ 1. printf'STIES? Η*-30s ιd\n' 'Obsolete'. Soptionsl 'cm.obsolete.threshold )l -f Soptionsl 'debug' I > 2 i eisif ( next I comment I Stun) , pπrtf IST.EP? "t»-30s %d\n'. Surl, SthresholdlSurll) if Soptionsl 'debug > 2) I c.oseiTHΛlS- -f I 'Sset.-etaulr. Scptιc.-.s cn__e;_tnreshold ) * iιstr2secs( Soptionsl ca.def.tnreshold" } ) -f 'Sset_:csolete ScFtitr.S ' cn_:cs3ie-.e. hreshold' ) ■ .str2secs Soptionsl cm.oosoiete. threshold )' Surls_r.rocessβd » 0. •new.ur.s » Jc.d.-rls > esjupped.urls • enev.ctiJuums > •old.chluums • •inaccessic.e.αr.s • enooots > I). print STD_R". 'Current time is ■ time. *\n' if Sopcions! 'debug' ) , i create pipes to talk to the server -- th s should be changed to use POST > directly pipβlSETURLS SB.TXJRLS I die 'Can't create pipe: S". p pelcrrKsr.TS SEMRKaLTSI || d e 'Can't create pipe S". ..' iSpid • fork) I closelCrrjRiSI closeiSBΦRεsULTS) 1 elsif idefir.ei Spid) ( close (S-SWP-Si closelCETRESUlTS) openiSTDClT '>43ENDRESulTS') || die 'Can't dup stdouf openlSTClN ^iGETORLS') || die 'Can't dup stdin'. exec Swnats -.eu || die 'Can't exec POST- S". 1 else { die 'Can t fork S" 1 Si ■ 0 for Surl ,keys lurls) ( local I St < .threshold(Surl) . next f 'St «» -1). printf STT-RP 'CHC.SRV Surl threshold Sc\n' if (Soptionsl 'verbose' )) next unlessiSurl «- /"http./ I next if iSur. — m'-http •\ 1i l I skip scripts printf (SEMIVSIS 'isUWΛd-tstsvn'. (Si > 0) ' '.' " Si. Surl) Si" ) tiose' SDTXRLS: -nilei<GϊTF^SVlT£> print.ST-ϊ?. 5 • <—£_ • f Soptionsl 'debug') > 1, next -f -.ittp caop (Surl die 'B -nosuc next loca.l sp Snceot -f 'S S " if Soptionsl verbose i ;..-er = ::i:r.e_crecιcec -f Scacne.cneciced > Swnen ScachedjnodcimeslSurl) > joints.. Smod. Swhen. Snobots). SscamplSurl) * 'proxy-caching server'. elsif i Smod > Ol ( prmtlSTDEWl Old info fcr Surl\n'l if Soptionsl 'debug' I > 0. • create pipes to tallc to the local back end pipβlCETORLS. SBπSlllSl || die 'Can t create pipe- S". pipelcεπtεS'JLTS. SEMDRESULTSI ll die 'Can't create pipe. S" if ISpid = forkl ( close(GETURLS) : ' ' 11 die 'Can t dup stdouf die "Can't i-p stdir.' Can't exec Sprocess.^rls. S" for Surl i eys lullsI I local(St: ■ ithresholdlSurl) . if (St »« -11 ( print STSERR 'Skipping ignored URL url r.* :'. iscpcionsl 'debug' )ι. next: 1 print TTOBUt 'CKK. Surl threshold St'.r.* if (Soptionsl verbose')) . next . les (Surl — /'http:/ || Surl •- /"fιle;/i. next f ISurl •- m'"h-tp:. , \'") ; t skip scripts last if (Soptionilmax.utls) >* 0 -. ♦♦ Surlsjrecessed > Sopt.onslmax.uzls) I ; Slast.checked • Smod • Svisit.timeslSurl). Scicned.cs • 0. if (Surl •- /'http: /I I if (defined Scached.aodcine lSuilll I IScached.mod. Scacbad.cneeked. Snobots) * split IS;. Scached.modcimeslSurl)): if ( Scached.cheekβd >< Slast.checked) ( Slast.checked • Scached.checxed: SstamplSuril - 'previous run': 1 if IScached.mod > 01 { if (Scached.mod > Svisit.timeslSurl) .. (time - Scached.mod <> Soptionsl 'c .obsolete.-hreshold') I 14 'Soptionsl 'want.latesc' ) I ( pushlenev.urli. Surl): Surls_-ime(Surll • Scached.mod: print STDEM 'Surl known to be new: ikippingvn' if (Soptionsl 'verbose' I ) : if (Soptionsl 'debug') I { locallSmdatel • .ctiaelSurls.tiaβlSurl)) : local ( Svdate ) : Svdatt > .cciaelSvisit.timeslSuxll) if Svisit.timeslSurl): Svdate > 'Never visited ' unless Svisit.timeslSurl). chop Sadtte: chop Svdate: print STDERF '\tvιsιtad Svdate (Svisit.timeslSurl)) modified Smdatevn* ) next: 1 Smod : scached.mod, ) elsif (Scacned.mod < 0) < Scacheα.cs * -Scached.mod. ) 1 if (Snobots eς toots') ( push lenooots.Surl) : print STDEW1 'Surl known not to aiicw robots, skippmgvn' if (Soptionsl 'verbose' )) . next: ) print STD-JUl 'URL Surl visited Sv sιt.tιaes(Surl) mod S odVn' if [Soptionsl debug' ) 1 , • store the max fcr later - used for knowing when to add emphasis Surls.checked(Surl) » Slast.checked. if (SΪast.checMd • St > time) I prmtf (STCKJ. 'Using cached s.r. ace VJ threshold »d ISstamp(Surl))l\n' , toe - Slast.checked. St' .; iSoptionsrveroose' I), if (Smod > Svisit.timeslSurl! pushienev.urls. Surl ) else ( pushlβskipped.urls. Surl) . I Sur Is. tune (Surl ! < Smod it Smod > 0 : next: I print STDERR "Checking Surlvn' if I Soptiens r verbose ) ) : •next if I So ions ( ' nop ' ) ) : • need tc change this to handle sussing files, down hosts, etc . — be I smarter about return codes . local lSsendαsd) « sprint. I 'is td ιd\n' . Surl. Slast.checked. I Scached.mod > 01 ' Scached.mod . 0 ) .- print SBUURLS Ssendemd: printf STDEM. ' --> Ssendαnd' if Soptionsl 'debug' ) > 0: - else { I file: local lSfile) • Surl. Sfile =- s'*file: '-'>.''>: Sfile * - s?-file: ?S-JJV( 'H01IE' ) ' . Sfile «- s'-/localhost?? . ;f ' -e Sfile) I localleeryi = scatl.) : Stimet « SarylSST.nTME): Surls.ti elSurli < Stimet. Sstamp(Suri) * 'locai fiiesystee'. printISTDERP. 'F le Sfile exists -> Stimetvr.'i f (Soptionsl 'debug' P . f (Stimet > Svιsιt.tιaes(Surll) ■ pushlenew.urls. SurlI : ) else I pushItold urls. SurlI : I ) else I pushlβinaccessible.urls. Surl): pnntlS DERR 'File Sfile does not exιst\n'! if (Soptionsl'det..')).- next. ) I ] cioselSQttWtiS) : while (<GETRESU TS>) { pnntlSTDERR '<—$_') if Soptionsl'debug') > 0; chop.- (Surl. Smod. Sstatus, Sresp. Sres 1 > splitlΛs*/): die 'Bad input from back end' if Srest ne " I Sstatus eq ": .nosuchurl(Surl). next if :defined Surls(Surl): if (Sstatus eq 'NOBOTS'I I push (βnobots.Surl) ; Scached.modtimeslSuri! ■ joinlS,. Surls.timelSurl). Svisit.timeslSurl). 'nobots'). next. : if (Sstatus eq 'HOK') ( SstatuslSurl) « Sresp if Sresp ne ": pushltinaccessible.urls, Surl); • don't check again until timeout • Surls.checked(Suxl) ■ time: Scached.modtij*es(Surl) • joinlS ; . Surls.tiaelSurl) . time) , next: f (Sstatus ne "OK') ( print (STDERR 'Warning, bad status Sstacus from back end. url Surl\n'l; I :f (Smod ' 0) I local ISchanged); :f (Smod <■ Svisit.timeslSurl)) ( pushltold.urls. Surl): Schanged * 0; i else { pushItnew.urls. Surl! , Scnanged > 1: ) Surls.tιαe(Surl) » Smod. Scacnedjuodtimesl url) » )θ n($;. Smod. time): printfISTDERR 'OK: Surl (Smod) ιschaπged\r.'. Schanged ? ":"un"I :f (Soptionsl'veroose' )) . SstaiaplSurl) » 'polled'. ) elsif (Smod < 0) I printfISTDERR 'ChecxsumiSurl):Scached.es -> »d\π'. -Smod) if I Soptiens I ' debug ' ) ) . Scacr.eo.aodcisieslSurl i = loin IS , . -Snβw.cs . time) . .: ( Scached.cs • « Snβw.cs I ( push I •new.chksuαs . Surl ! . push I .old.chlcsums , Surl ) . ) :'. 'Soptionsl verbose I' •exit .'. 'Soptionsl 'nop' I) print OUT <html> '\n', .'. (Soptionsl 'display.header' 11 { thoplSdate < -ctimeltioe) ). print OUT <<E0F <tιtle>Status of UiUS</tιtle> «"..>Tπιs page generated by <a href * 'Sw3newer_ir.'>i3r.ewer vS-3ne-er_ver< a> on Sdate</h6> -OF ) print our -display.url.list .: Soptionsl display.footer II I print OUT <<EOF <nr> t'.:s page generated by <a nref-'Swjnewer. ' .rl'>wjnewer vSw3nβwer_"er</a> i crogram -rirten DV S<jnewer_author and mooitiec CΛ Swjnewer.aut.-.orϊ EOF ) print OUT </html> 'Nn* t.oseiOUTl fl.save.checKsu if (Soptionsl checksum')) .: (Soptionsl cache.πodcimes' )l ( iwrite.cached.modtimeslSopt onsCem.file i scacπed.modci.' exit sub url.desc I local I Surl) » •_ if ISurlslSurl)) ( return Surls(Surl) ) else I return Surl s b Dydesc ( .url.desc ( Sa i <■> (.url.desc I Sb) , sub d splay.url.by.monrJi I local leurllisc) • »_. local (Surl. Sold.month. Smonth. Smonth.no. Slist . S_. Sday, Svday. Swdaysus. Sbold) . loca ( vdate . S ource ) . Slist Smonth • " . for Surl Iturllistl I iSurls.tijπelSurl))) I3.4.5,.! , Slist • .url.desc I Surl) Slist - *</a> I" Sbold • i Soptionsl 'embolden' ) .. (S.rls.tιme(Surl) > Svιsιt.tιmes(S rD) .. (Svisit.timeslSurl) > Ol) .'. (Soptionsl last.visited' I .4 Svisit.timeslSurl) > 0) ( chopiSvdate ■ .ctιaβ(Svιsιt.tιmeslSurll) I Svdate = ' visited Svdate* I else I Svdate ■ " -f (Soptionsl timestamp.source I -. SstacφlS.:!) ne " Ssource = based on Sstaπ lSurl) ' else i snapsner S O snapshot ' local ISurl! « ... return " :f Sun '- "http./. f :SoptiensI' orms' I) I return qς 1 <fcnt sιιe*-3> cfern letnod'POST action * 'Soptionsl snapsnot.url')*> <ιr.puc .name * ' rl* type * 'hιdden' vaiuβ * 'Surl' > <ιr.put name * 'emai ' type*'hidden' value*'Scptions! 'frtr.acdress 1"> <sβiect name * 'type'> Sopcion.string <.selecc> < nput rype-'suom t' value * 'invoke** <'form <ι font> i else I local ISa) = qq'<A HSEF= * Soptιons( 'snapshot.'.rl' l'emaιl*Scpt cr.5 • f rcc.addres! !-url * Surlitype * i ; I'll, return cjq 1 | S(alRemesjιoer'>Reaβmber</A> | Sla)Oιff>Dιff</λ> ■ 3,a)Hιs::r/'>)l:story</A> •: • '. suo I tnew.urlsl if iSoptionsCchecksum')) ( if (•new.chjuurω) { Slist . * *<U>Tne following URLs may have changed Ino modification tune available) :< h3>\n<ul>\n'; foreach Surl ( βnew.chksus ) { Sdesc • i rl.desclSurl); Slist .» c-q?<lι><"ioβr> <e href * 'Surl*>Sdesc</a>\n?: Slist > -snapshot I Surl! if Sopcion r napshot ') , Slist • *<'rιobr>*.- ) Slist * *</ul>\n*; ) if llold.chksuasl { I Slist * *<h3>The following URLs don't seem to have changed mo modification time available) :</hJ>\n<ul>\n' foreach Surl ( β old.chksums: { Sdesc • -url.desclSurl); Slist .» cjq?<li><nobr> <a Slist .* (.snapshot I Surl I if Sopcions c snapshot ) • Slist * *</nobr'*: 1 Slist = '</ul>\n'; if 1 Soptiens ( • display.recent ' ) .- tskipped.urls ) ( Slist * '<h3>The following URLs were checiced recently and not p:..ed < for Surl (sort bydesc tskipped.urls I I Sdesc * (.url.desc I Surl) : Slist .= qtj?<lι><nobr> <a href*'Surl*>Sdesc</a>?: if iSoptionsl'tiaiestaiTjp.source' ) .. ssta-cp'.Surl) ne "1 ( local (Sage) * time - Surls.checkedlS.rl) ; f (Sage < 01 ( if (Soptionsl 'verbose') I ( locallSfoo, Sfcol. Sfoo2). choplSfoO'lctimei time) ) . Sfool * Sfoo: choplSfoo*-eC!me ( Surls_chec>ee..S.r.: 1 Sfoo2 * Sfoo warn 'Surl apparently chected n the future' printf (STDEFffi "tine *s c.-.eciced ts r.' Sfool. Sfoc2>, Sage .ocal i ci * isecs.scrlSagei localisagoi • spnnt fis Is' (SstaβplS.ri es, • hiscory' 1 ' ■ visited' • t.-ieeked" St! S. s: = ' based on SstaoplS-rl Sago ago)' :..; • = «sr.apshct:Surl .'. Soptionsl sr.apshct'). .'. S;;:_;-.s d sp.ay.-ncπanged ■ -• iold.urls S..S-. -■ • <hj>τhe follo'.'ir.g '."F-.s have not :.-.anoed < ; h3><r." i.-s: * icisp.ay.jri.cy.mcnth' everse set: S r-s.t -ftβ*Saι<*>S.rls_t aβ(S. »old_.rLs) .1 '::t :r. l- ' .s: '.-.-. S'.r Sdesc • .url.desc(Surl). Slist * ςq 7 <lι><nobr> <a hrβf*'Sur ">Sdβsc< Λ :Suris_accessedlSurl) > 0) I c.icpiSvisit * .ctimeiSurls.accessedlSurlli I . Slist * 'llasc visited Svisi l ' I Sli • enobots) Slist * *<h3>The following URLs are not accessible to robots foreech Surl ilnobocs) f Sdesc * turl.desclSurl) . Slist ■ ςq'<lι><nobr> <a h^ef* ■ Surl • >Sdesc< a>\n , ; Slist * .snapshotISurl) f Soptionsl snapshot' ); Slist • *<cnobr>'- 1 Slist * '</ul>\n'. returniSlisti , ■ a generic routine to figure out what k nd of hotlist we're looking at • potentially useful if the hotlist s fed m through stdin or something sun load.hotlist ( locallSfiie ' oclist.urls 'hotlist.dates) * ». .ocalitidescs scimes.S.I. cpenCN.Sfile || die 'Car.'t open f le Sfile for reading S'\n' S. • <IN> :'. l ncsa- 'rosaιc-hotlιst-foπnat-1/l ( -load.mosa c.r.otlist.or.hiscoryiπi 1 'hotlist.dates 'hotlist.urlsi ) elsif ' '<'30CTΪP. IMC.HI ET--λPE)--ooιcaark-f le-l>/ ) ( .load.netscape.oooioaarkslIN 'hoclist.da.es 'hotlist.urlsi ) else < die 'Don't know how to handle hotlist with format S_' . ciaMl-Ki :ao_-xs.::_.-..clis;_:r_historv teal IS.-.-.-.;.. Shot. st "tiats '.escs • »_ teal's :.:. Sdesc stine- 'Detau. -./Global' 'Co.- t parse input line S.': :: i.-.ttiist :.-.:piSdes: * <Shandle> ) : itestsiS.rl) = Sdesc. I i::τ=i Surl. = ilocaldatetomtunelStmei :ai_:. * -.i:.:e_c:cιcrjιrks I :ta. :.-._.-.-le 'ti-es 'descs. ::s. :. :_:l. Sdesc Stimei . ::a. :...-..'.< 5i r.<s, . fl discard 'Default* r * . ■ ■ <Σ.-.i.-.dle> • rest of f le .--.-. '.ir.ics ;.om(".n . βconcentl I . zi-.:τ.ez Sl-nks('*-SlιιucSι I * S.ir.sslSlmks) ; £u:l • S . Sdesc • " ; .:' ..-AST.VISIT*'! • )* ) ( Stime • SI; ■ else 1 Stiae • 0. • r. le idefined Slinksl — Slinksl) I last if iSlinkslSlinksl •- Sdesc * Sli kslSlinksl : cπr.t STD-Wt ■ extract.links found UKL Surl tune Stimβvn* if I Soptionsl 'debug' ) > 51. jcescslSurli * Sdesc; StisesiSurli * Stime: e.se '. cr.r.t STOtM * extract.links ignoring S.'π* if I Soptions (' debu ' I > 5). r.ext I "Jittc fcr .-.isccry files, sue .ca-i..-. stcry ( lOcaliSfi ei » β., .:cal.t: =es.S_.. tcenc'.Sfile 1 || die 'Can't open file Sfile for reading : S!\n": trir.t ' ad r.5 history file '; l_ * < * _.>, .: !/r.c∑a- '-nosaιc-hιscory-forβat-1/1 [ .::i:.r.csaιc_hotl st_or .history I IN.0,'tιmesι ,- e-ϊi: -l-ri-.-Glctal-history-file-l/t I >. ι ^> = .icac.r.etscape.nistoryiπil ; ,::ι. Zir. * Scptionsl 'debug' } " ■ "die" 'warn': eve. i.e. • ϊen't know how co handle histcry file Sfile with .:ta • '..vs itises) ' cc.Sneyi = 'history' local's. Surl.StiMl : .h e < <Shandle>> I cncp. if l/l\5-l>s*l • ) /) ( Surl • Si. Stime • S2 i else I die "Can't parse input line S.'. \ S inesISurl) • Stime return Itimes suo extr.:t_li.-.ics.desc ( ..cal'S.n..r. * sh ftlt.l loca. t.mxs.tlinksi .ocai eret S.:. i_ Sdesc. ineaαers.Sconcent.SrespcnseI Sresponse * •.v. lreq βstI 'CΪT' ccntent Sgl.request. iaeouti , I' -f Sept ensi 'debu ' )) I "mεKK ■ — www content ST-:!?.?. Sconcent. S7__?-"t "\n end WWW content- •parse.ht.-: " 1 inks . Sconcent I Slinks * -1. while defined SUnksl—Slinksl ) { l 'debug' I > 5): debu ' ) > 5). if 1 Sopcions I 'debug') > S), return i Iret I sub ; Sresponse * ιwwv'1 request: 'GET', -ln.url 'headers 'concent, Sgl.request.timeouti • .parse _stal i 'links . Scontent I . S limes ..-.lie .defined Slinksl'-Slinksl ) I IS linxs I Slinks i »- I ISt l oe' ■ i-www'statlSl.SUserAgent SReplyToll 111. •' uniess istune) I pushleret, 'Unknown' I. next. ) iSd J y. Smontn.no, Syear,Svdaynu*t,ι * :gmtιmeιStιs_ι ) 13.4.5,61. Smonth * Smonths I Smonth.no 1 , S._ay * sdays I Swdaynuml . pushleret. 'Swday. Smonth Sday Syeaf else I c-shUret.SlinkslSlinksll sun reac.cacnes.-ftodtimes ( local I Sfile) « «.: local fitiaes. Surl. Smod. Scnecked. Snobots) opemCλCHΩ.Stile || return il: while KCACHED> " ) ( (Surl. Schecked. Smod. Snobots. Srest) * spiiclΛ die 'Bad input from Sfile' if Srest ne " St meslSurll = -.oiniS., Seed. Schecked. Snooocsi SstamplSurl) * 'previous run': ) close(CACHED! . return itines. sub close, A3!ED>. localdatetoatimel ) s a modification of the gec_o-ιc mefrom wwdates .pl Description Translate a date string to machine tune i seconds since Epoch) Usage . Smtime * .localdatetomcimelSdatei wnere Sdate in in the following format • Thu Feb 3 11 .03 55 1994' sun localdatetomtime localIS.) • «_, local(S[) < 0: locallSday. Sun. Syr. Shr. Smin. Ssec. Sedate. Satime. Soon. Smidxl: locallSoftsetl « 0. I Split dace string locall»w) * split. ( S eekday. Smn. Sday. Sati e. Syr) • tw; Syr -* 1900; IShr . Smin. Ssec) * split ! / .' . Satime) : if l !Smn | | ISyr < 70) 1 I return 0: ) • Translate month name to number Sαudλ * indexISMstr. substrιsmn.0.3 ) ) . if ' Smidx < 01 ( return 0; ι eise I Smon = Smidjc i ) ; I ' Translate to seconds since Epoch return i.cimelocal tSsec. Smin. Shr. Sday. Smcr. Syrl ) . i'/usr/local/bin/perl I This script is a bacicend Co the snapshot script m my CGI directory » It is separated primarily for purposes of debugging it doesn 't rely • on having a CGI-environment set up and insteao worxs just with l cciroano-line ergs • One of the things being done for simplicity here is to j-ust keep » appendin; to the iog of <ur..c.-ιestamp> tup.es. instead of t overwriting n place. • to do- • locking Sdebug: 1 Snees.loeics * 1, Smaxlength = 1024 • 511 t 512 Kbytes Ssnapsnst = 'http /wwwil2ti research. att. com/cgi-bin/schaods* i this snould be PWD b.t that may not be normalized tc /home Ssnapαir = ■ /homβ/dougiis/tap/cg test/snapshot* I locallSoldfhi = se.ect (STDERR) , Si = 1. seiec tSTDOUT) S|= 1 select ISoldfh) unsni- I9INC. ' ,'home/douclιs/ lib/perl* ) . t arcm.ecture-specific library (for sys/socicet.pn) require 'arcb.pl*. Sarcn = -arch; if l-d iSlibloc = vhome/douglιs/arch/Sarcb/ b/Ferl * l ) ( unsniftlS NC, Sliblocl; ) • ISFD.CLQEXEC. SF.SETFD) = II, 2), I from solans sys/fcntl.h Sllb.www.ver = '0.40' unshiftUINC. SENVi'LIBWWW.PER ' I || '/hc«e/doug s/ b/perl/Ubwww-perl-$lib_www_ver' , I Sroot * SENVl 'PWD'). if (defined SENVl 'PATH' 11 I « print STDERR "Path: SENVl 'PATH' )<br>* if Sdtbug; ) else { SENV( 'PATH' ) = ' /home/ ouglis/axcb/Sarch/bin./home/douglis/bin: /bin/. /usr/Sbin./usr/bin./usr/ucb*. ) unless (SENVl 'http_proxy')) [ • SENVl 'no jroxy')='4tt. con'; > want research.att.com to go outside SENVl'no.proxy' )=*tbu.att.com,ncr.com.radisn.research,att.com.wwwll26 research,at .co .ih.att.com,ho.att.cca.cb.att cos.nt.att **" ■ t-com.divlll.att.com*. SEN I' ttpjroxy )='http • //radish.research.att.cam:80007 * , • SENVl 'httpjroxy' )=*http //bluel.rβitarcn.att.com:8080/', ) require 'cg -lib.pl' SEXIT.FN = 'cleanup.tap' require -cgi-exit pi' require 'cgi-aiarm pi" require 'duplexpipe pi * require ■ nonaaiiie_html.pl * require 'ctime.pl require scat.pl require "www.pl*, require *lock.pl* ..' Sneed.locks. •require * lush.pl * require *prιntιd.pl*. . i 'Scebug) I Sargstrmo - -om' HAFGVl print ST-ΕRR -printic require • mdprog.pl": KSrcsdiff .Shts di£f.S£ilter_anchors)='.findprogl * rcsdi£f. 'htmldiff'. 'filter.ancnors'l ; iSrcsdiff . -htmldif .5rlog2htal)=i£ιndprog ( Tcsdif f * . "htmldiff . *rlog2html* I ; -exιt_3race£ullyt 'Unable to locate rcsdiff. Ol if idefiπed Sresdiff; -exit.gracefully I "Unable to locate hcaldiff. 0) if idefined -htmldiff; .exit.gracefully ("Unable tc locate rlog2html * . Ol if Idefined Srlog2htal; •J.exιt__racefully I 'Unable co locate filter.anchors", 0) I if 'defined S filter.anchors; Srcsdir = Sresdiff; Srcsdir =- s'/rcsdiffS! ! , Scι= * Srcsdιr/cι * . Snc.nar.is.url = • http:/,wwwl!26.research.att.com/-douglis.no_nands/ * Sno.r.ar.cs_autnor = <<EOF; <A HSEF=*-c_p- //wwwil.6.research,att.comc-άougi s/*>Freά .ou;ii5</A> EOF cnoplSr.c.nands.auchori ; Sncmlcι::_author = <<EOF; <A H?E"i'-t:p:/.'www-spr.ih.att.coin -tball '>Tca Bail</A> EOF c.-.o IShcr.-άiff.autnerI ; i .exic.gracefuilyl"Wrong numoer of arguments'. . ') if SIARGV '= 2; l need £_.l path here because of possible BASE directive Slogo : qq'<a nref="http://ww ll2G.researcn.act.com/-douglιs/no_hands/> <ιmg al gn=*leff alt='|U0 HANDS logo)" srcs"hccp://wwwll2..research. tt.com/-douglis/no_hands/logo.gif • x'A»Vn'. ; Strailer * «EOF; </BODY> <hr> <FONT SI_E=_> This page generated by <a hre£- ; "Sno_haiids.url">NO HλHDS</a>. an <Strong>AT_aap;T propriecary</S.rong> program written oy Sno.hands.autho . toucan. research. att. com * >comments</a> axe appreciated. (also proprietary, -rit*- • ISoperation. Suser. Surl. Suserversion) = β λXGV; etspfiles - I) ; ed fields. u->URL. e->email(userl , ew * , , '.lst.Al.', "e", 'History', "ue" , • Users', • ? * , ■ Rβaeroer. f.new' , 'uew' , ■ AU.URLs', * p * . 'URL.users' . "up" , 'Viβw.Current' , "ue" .exit.gracefully:"Illegal operation Soperation specified". 1! .mess defined Saliowaoie.operations(Soperation); « Fris STDERF *\Sallowable_operations(Soperation)=\*SallcwaDie.operations(Soperationl\*\n* if Sdebug; f Sa ' .ς.a D ie.cperationslSoperacion) =- it') ' -exit_;racerul- . y('Illegal user name*. 1, -f Suser - "'\w*.l«S 3t:re * = -people/Suser * ! . 00.23, 1995^ /home/tball/douglis-tmp/nohandsBE :: ,Sailowaole_operatιons(Soperatιon) »- /u/l I .exi:_;racefuily(sprιntfiqqϋllegal URL: "is"', Surl). li if Suri !- -i!"hCtp://|\w..-.Λ-:|»S!: > cer.; r.t . add logging i: (Seperation eq • View.C.rrent") ( print "Location: Surl\n\n*; ex t; Sr.eec.rcs = 1; .• else ■' Sneed.rcs * 0; :: Sa.l-warie.operacionslSoperation) *■- /p/) ( oper.cir(PEOPLE, 'people' I 11 .exit.graeefuliyI'Unable to open directory people '. 0), .-caillpeopie) = readdir1PE0PLE) ; " tises = grepl/'Ia-rl/ -4 s '. " 'people.' ! . ipeoplel: printf STDERR "Checxir.. users »s\n", join; ' ', βtiaesl if Sdebug > I :.oseIPEOPLE) ; .: -e Srcsfile) ( printf STDERR "Is previously checked ιn\n*. Surl if Sdeoug > _. ■ -exιc_grace£ully<*Fιle Srcsf e noc wπtasle'. 3! unless -• • Srcsfile: Sexisted > 1: else I Sexisted * 0: I ' users - I f Soperation eq '."".-.users*: f "Soperation eq "Diff* .. Suierverjion !- /\d»/l || Soperation eq 'List_All" | | Soperation eq 'History* || Soperation =- /."users/1 || Soperation eq • Rememoer.if.new' || Separation eq 'λll.URLs") ( foreacn Stimes (Iciβesl ( : (Soperation ne "URL.uitrs'l ( undef Slastseen; undef tlastseβπ; if i-r Stimes) ( .ex c.gracefullyl'file Sroot/Stiaes not writable*. 0) if ' -w • stimes" openlTIHES, "<St mes'l || fctxιt_gracefully("Unable to open Sroot/Stimes: Si*. 0); unless (!Sneed.locxs || .sloe):(TIKES, 1)) ( -exitjracefullyl"Unable to acquire lock on Stimes*, 'λ file was temporarily unavailable. Please try again later. (You can reload this page ι* ) while (<TDt_S>) ( next if /*$/; local(Sturl, (when) - splitI' '. SJ ; if (Soperation eq 'D ff* | | Soperation eq 'History' | | Soperation eq 'Raaenbcr.if.new" || Soperation eq 'URL.users') ( f ISturl eq Surl) ( f ISoperation eq ■ URL.users*) ( local(Suser) - Stimes; Suier ■- si"people/!!. • Slastseen)Suser) = • βwhen*; ) else < Sprevseen * Slastseen if Suservers on eq 'penultimate'; Slastseen - 'Uwhen"; - • Slastseen = join " '. Iwhen); ) ) i elsif (Soperation eq "List_A.ll" || Soperation eq 'Users' || Soperation eq 'All.URLs'l ( SlastseenlSturli = 'Iwhen*; > SlastseenlSturli . -oinC '. βwhen): if (Soperation eq 'All.URLs'l ( t set here; use if set "use lx SlastuserlSturl) - Stimes: Slastuser(Sturl) =- s!"people/' ' : ) Slastseen = Sprevseen £ defined Sprevseen _• Suservers on ec *penu.t.r.ate* close TIMES) -unlock(TIMES) if Sneed.locks .f ISoperat.on eq "users') ( local (βkeys) = Keys ilastseen local(Suserl - Stimes Suser =- s "people/ ' ScourtlSuseπ = Slkeys • 1 1 els.f ISoperat.cn eς * All_l"RLs * > far Ikeys slastseen) ( Scount($_)• ♦ ) i eisi: (Soperation eq "Users"! ( pr.-t " otn -g has ever oeen checxeα in * pr.rt Strailer exit C ) .' Soperation eq '(.lst.Al.' l ( pr.nt iT.tie rpages recorded lor Suser* pr.-t * <n*,>Sortea oy ■ pr.-t Slogo pr.rt * <TA3_E> <TR> <T u >("Sl,</TH><TH>Mcc-: cat.c- t_mestajp</TH>< * "R>\-* fcr Iscrt -yαoωin Keys ilastseen) ( iocs. Sur. Irest = printf qq <TR χ ""_xA Ssnaps-c: Sur. Suser l l S.astseenf S print '</TAB_E> * Straner ) elsif ISoperacion eq 'Users') ( print iT.tlel-Users of the NO HANDS service") print '<-AB E> < * "R> <TH>User</THxTH>Pages fcr (reverse sort byusercount keys ".count) I printf qq <TRxTDxA HREF- s'e a.ists- y e^ istit.OAll'JS^ A> < i'Z^ -Z>\ά< * *_>< Ssnapshot %_ Scsu-t(S. ) print "</TAB_£>" Strailer ) eisif (Soperation eq 'URL.users'1 ( pr.nt -TitleI'Users wno nave recorded Surl") • print Slogo for isort xeys Ilastseen) ( printf qq <A HREf-**ιs' > emaιl=*sl.type=Lιst"ι"ι20All">S_<Α> (»sl<-R>\r Ssnapsnot S_ slastseen!S_) ) print Straner ex t 0 ) elsif (Soperation eq *All.URLs") ( print -Title("Pages recorded by the NO HANDS service") I print Slogo print "<TABLE> <TR> <TH>URL</TH><TH>Userls)</THxTH>Ijast versιon</THx/TR>\n' for Isort oyuβercountUU. keys tcountl ( printf qq'<TR><TDxA HREr-="%β?url«%s.eι-aιl=»s-type«Vιe %"ι20CLrrent">»s</A></TD><:TD> Ssnapshot S. Suser. S_ .f (Scount(S_) > 1) ( printf qq'td <λ HEEF='*ιs , url*tsitype=UR *ι*l20userjte_ιaιl*ls'>users<,'λ> Scount15.; Ssnapshot S_ Suser ) eise ( printf qq'<A HREr='t,s'type=I.ιst»%20AlUιemail-%s">»s</A> Ssnapshot SlastusertS.i Slastuser(S_) ) pri-t *</TD>* localisfn) = ircst.-el.urlZfnlS.l) getvers.on .£ -e Sfn) ( openlRCSFIL; Sfn) |j warn "Ca- t open SfV last getversic- • do we neeα to lock jus to read 1st line' locaKSline = <R.SFILE> .f (Sline =- -head\s»(\d»\ \d»)/l ( print: qq <TDxA ffllEF=" .s' , url=%sie ails*s-type=Histcry' >is< -, χ /TD>\n Ssnapsnot S_ Suser SI closeI CSFILϊ 1 ) prirt *</TR>\n* p-.r" "< TAB-E>\nStra.ler" ex.- 0 e.s.' Soperation eq 'le-emoer-if new" :-' defιr.ed( Slastseen) ) ( Scsg = <<E0F; This <A KSZ-=* S ' .rl *>page< /A> was previously saved. <A HREF: " Ssr.aps."icf , url=Surl-email=Suserl.typeiHistcry* >View the -ustory« A> ECF pπr.-- S.τsg.Strailer; ) eis : . '.eϊmedlSiastseeni .4 Soperation et; *Dιff: ( a snapshot</A> tne hιstory< A> EOF iexii.-jracefull ruser tried to access a UR_ not previously seen * . S s j . 1 elsif -• Srcsfile; «exi:_;rscef..l/'"T.".e RCS history asscc ate with the -π <a is unavailable*, ll, « i.:.. " ".-.ea.ers.Sccnter.t.Sresponse) ; ' later. (You can reload th s page. print STDERR Iprintid. 'Acquired lock on Slockfileλn* f Sdebug; ) if (Soperation eq 'Diff || Soperation =- /RemβmDer.V) ( Salivepid - -cς;_κeepalιve; print STDERR iprintid, " eepal ve p d Salιvepιd\n' f Sdebug; Snewuri = Suri; Sresponse - -www'lrequestl'GET' . "newurl. "headers, 'concent); I' kill I. Salivepid; I SIGDfT if ϋwaitpidtSalivep d. 01) ( printf STDERR *%s warning: unable to wait for child keepalive process SalivepidNr.'. Iprintid; if ( Sresponse '- I'll) { localISrest) = Swwwerror'ReβpMessage(Sresponse); I'; if (defιned(Srestl) ( Srest - • : Srest"; ) -exιt__racefullyI'Response Sresponse during GET requestSres . ■ , 1); ) local (Slen) = lengthISconcent) ; i- S ert > Siraxlength) ( ■exi:_;racefully;*The page is coo large to cacne. Content length is Slen; maxiour: is Smaxlength. * , 1), crir.::: _?_•_ * _ ■ . 'Get worκed!!\n content:\n%s...\n', -sucstitute.eπtities (substrISconcent. 0, 25διP if Sdebug > 1; Scsnter.t = tncraaliientaKSconcent, Snewuri); -: Sccntent -- /"Errcr filtering HTML:/) ( -exit.gracefullylScontent, 0); open .'.Turn"?. '>Scurrent'l || .exit.gracefully('Unable to open Scurrent', 0) PLS-'it=p ilss. Scurrent); ct 23, 1995^ /home bail/douglis-tmp/nohandsBE if -: "Sc_rrer.fi i ..ca.'S.e.n, . "S.er. = ler.gt.ήlSconter.tl ) > 0) _exit_;raceiuily("Content r.ct correctly copied tc Sconcent, length Slen" 0). -exit.cracef-liy "" "Page is apparently empty"); ) pr.r.tf -TD_ := ■* !«*- * Iprintid. Is -1 Scurrent' if Sdebug; ) .: ( Soperiticr. e * .iff* ' oper. :".: *>.;.rrer.t.snap') || -ex-c_;raca:_l./ "πafcie to create * Sc_rrent.snap\"" , 0); P-s". ϊt~:..es *Sc.rrer.t.sπap * ' ■ if S.server.icr =- sat) ( i.astsee- = " -rS.servers on * 1 e.ss i.astsee- = ■ -_ i.astseen' ■ s.istsee- = '-_' Clastseen Scr-o - c'.p.t = .c.r.ejcpipeiOLT "IS • _.?_ ("See" *-p*. Ilastseen. 'Srcsfile') I ' . ■ Sere = i-.z = .ccex ipeCCUT. • ::: • _?.= = '*, home'douglis/bin/testce* *-p" "Srcsfiie"! ) ' FΓIΓ.:: ΞTDϊ?? *_va..ac..n; cc =and %s\n * . Seed if Sdebug > 2; eva. Sc_ . if .SS) iaxi:__rs:-:ι:. /IS*. 01. I wni e . <IN> . print -LD print STD- K 'S " if Sdeoug > 2. ) print STDESS " .-.' Sdeoug > 2.. S. > 0. close IN. print STDERR .printii. "Error S 1 from close(IN)\n' if S? -- Sdebug; close CLD. closelOUT) , prir.t STDERR iprincic 'Error S' from ciosβlOUTHn* if S? .. Sdebug; if I Sdeoug) locallSfirsci = 1 -r.i.e(<_RRS>) i pri.-.t STDERR iprintid. 'Messages from co:\n', undef Sfirsc if defined -first; pri t STDERR ) close IERRS). print STCERR .pππtic. 'Error S' from close(ERRS)\n' if S' ii Sdebug; if CwaitpidlSpid. 0)1 ! print STDERR iprintid. 'warning: unable to wait for child process Spιd\n". ) elsif IS') ( print STDERR iprintid. 'SO- warning: co returned status $?\n'; I eisif iSdeougi ( trim STDESR -prir.cid. *S0. co returned OK status\n*; ) if (-: *Sc.rrer.t.snap*l ( -ccallS.s = '/cm/Is -1 "Scurrent.snap*' •exit.cracefullylS.s "Snapsnot o: page is apparently empty*). if '-« *Sc.rτer.c.sr.ap' — -s 'Scurrent* «. 's'stec "_ι.n/c".p' '-s", "Scurrent. snap' 'Scurrent')) ( print 'V: differences -Cilfs = J ) else < sdiffs = : Sa.ivepic = ic?i_.ceepaiive. prir.t S -I-vS tpri ti.. 'Keepalive pid SalivβpidVn" if Sdebug; itrai.er =- s. .ritten by Sno_n_nαs_-_t."θr) \ SlSh_Bldiff_craιler / :;*- :: rr ■ ∑ cr-c.f : Scurrent. snap ic.rrer.t Scurrent. iff I') || .ex.t.crace: "Jnaole to invone r.caidi.f. 1, pus.-.i.tnpfiles. "Scurrent.diff • ) ; if ISdebug) ( ..callSfirsci = 1; wnιlel<DIFF>l I psmt STDERR iprintid. 'Messages from htmldiff:\π*, undef Sfirst if defined Sfirst; print STDERR; ) if Sdebug> 2) I printf STDERR 'Calling closeIDIFFI\n'; CicselDIFF); if .SdebugI ( print STDERR iprintid. "Error S' from cioselDIFF)\r.* if S': elsif (Sdeoug > 21 I print STDERR iprintid. 'closeIDIFFI OK\n' , '..ll :, Salivep.d. I SIGINT i: "»'aιtpιd(Salιvepιd. 0)) ( print STDERR iprintid. 'warning: unacle to wait for cmld xeepalive process Salιvepιd\n* ■ .: ' -e 'Scurrent.diff) ( if ι-s .) ( Sconcent = 'cat Scurrent. iff' . Sconcenc =- s'</hcml>! :ι; print Sconcenc; 1 else \ princ "lo differences encouncered<br>\n'; Sd ffs 0; ' else ( iexit.gracefullyl'No diff file Scurrenc.diff creaced", 01: } I if ISdiffsl i Strailer -- s/ age/modi ied page/; ) Srnove- = qq! (No e that the URL previously provided. <A HREF="Surl">Surl</A>, moved here.)' if Snewuri ne Surl: Smayoe * - You snould view the current version <scrong>dιreccly</scrong> if you want <a href= * hctp. /wwwll26.research.att.com/-douglιs/cracκ_urls/ * >w3_ewer</a> to Know tr.e page was seen. <iι> <λ KREF-'Ssnapshot?emaιl=Suseriurl:Surlitype.Remeαber * >Rema_ιber curreoc versιon</A>.<br>' uniess Suser eq *cr.e.r.anges* Strailer = «EOF: <ui> <lι> <A HREF=*Snewuri'>vιew current versιon</A>. S ovβd 5maybe <lι> <A n?.EF=*Ssnapshoc?e_iaii=Suseriurl-;Surl4type=H scory>See che version hιscory</A>.<br> </ul> </HTML> EOF print Strailer; ) elsif (Soperation =- /Remember."/) ( if (Sexisted) ( if (Sdebug) ( open ISAVEERR. *>i_T_BIR") ; open (SAVEOUT. *>iST_OϋT") ; pipe (RCSERRS. RCSOUT) || iexιc_gracefully("Can't create pipe". 0): close(STDERR) ; open(STDERR. '-"iRCSOUT"); openISTDOCT, *>iRCSOUT"); > eise > fcr.t tSTDOUT, SF.SETFD. SFE.CLOEXEC) |! iexit.gracefullyt'Unable to set close-on-exec flag for stdou , 01; fcntl.STDERR. SF.SETFD, SFD.C OEXEC) || iexιt_gracef.lly("Unable to set close-on-exec flag for stderr", 0); sysce-irSrcs -1 Srcsfile*) ii iexit.gracefullyI"Unable to Iocs Srcsfile*. 0); .: : deoug: ( loseIRCSOUTI; Close! TDER I; C-osel TDOUT) ; -per.IST-ERR. *>iSAVEERR" I ; :per.(STDCLT. * >iSAVEOUT * 1. -.ose.SAVEERR) . local ISwarn) = 0 wnne !<RCSERRS>) ( .f i'Swarn) ( print STDERR iprint.c 'Message from res Swarn = 1 ) print STDERR 1 ) else ( fcntKSTDOUT SFD.SETFD 0) ι| iexic gracefullyt'Unaole :. clear close-on-exec flag fcr stdouf 0) fcntl(STDERR SFD.SETFD 0) | iexιc_gracefullyCUnabιe to clear cicse-on-exec flag :cr scderr" 0) ) ) temp = CSC 'Scurrent' if (defined Sheaαers' lasc-acoifiec )' ι Sdate = 'Sheadersi lasc-ooαiflee )' • cneck fo in I Last-acc if iSdate ) 1 Sdace = 'SI S2 S3 1.S4 S ' ) splicelβtrp 1 : '-dsdace'l 1 else < Sdate = ict.se'ti e) cho ''lacei ) • noc clear if ./e really neec to send data to res -- leave as-is for now t Scurrent. =- s/ ( (*\tt\S| )/\\$l/g I local(Scad) = qq'\Spιd = idupiexpipe(OUT "IN -ERRS. \"Scurrenc2\") • local(Scmd) = qq'\Spιd = idupiexpipeIOUT *TN 'ERRS, \»tmp) ' printf STDERR 'Evaluating command ts.r." Seme if Sdeoug > 2, eval Scad, if l$β) ( lexιt_gracefully(Sβ. 0) ) if ('Sexisted) ( local(Stltle) printf OUT "This is a snapshot of a page, URL tιs\n", Surl t would De simpler in perlS if (Stic.e ' ihtml-tιtle(Scontent)) ( printf OUT '<brxι>Tιtle »s</ι>\n* Stltle ) ) else ( printf OUT qq'A snapshot made by <a HREF=*Ssnapsnot , e_ιaιl=»sitype--is-._All>%s</a>\n' Suser Suser ) ciose(OUT), local (Swam) - 0. Sunchanged * 0 while KIN>) ( if I debug) ( f CSwarnl I print STDERR iprintid "Message from ci stdout \n" Swarn • 1 ) print STDERR ) if ('"f.-e is unchanged reverting/) 1 Sur.cnangeα = 1 ) ) Swarn = 0 wnile KERRS>) ' -f (SdeougI ( .f CSwarnl < print STDERR iprint.d 'Message from ci stderr n' Swarn - 1 print STDERR loseCN'; CioselEPRS et 23, 1995^ home/tball/douglis-tmp/nohandsBE if MwaitpidlSpid, 0)) ( print STDERR iprintid, "warning: unable to wait for child process Spιd\n"; ) else < STATUS: ( local(Si) * S?; -1[2I0]S/ ii do ( I an ok status? princ STDERR iprintid, "warning: exited with a status of S?\n* if Sdebug; lasc STATUS; 1; /"OS/ ii lasc STATUS; iexic.gracefullyl'ci exiced with a scacus of SV. 1): ) ) iexit.gracefullyCProblem with 'times' file*. 0: if Slti es ι- 0: Stimes = Sti-ieslO] ; i iexitjracefullyI"file Sroot/Scines not wr taole'. Oi if ' -w 'St_ir.es'; if (Suncnanged) ( Sstatus - 'Note, file was unchanged from previous c-.ecr.-in.". ; else ! Sstatus = "Check in was successful ' "SJiscc princ Scraiier, "<p>"; c, nITIMES, '»$cιαes'l || «exιc_gracefully('Unable cc cpen Sroot/Scunes: '!', 0); _n,ιss ϋSneed.loeks || llocklTIMES, 0)1 ( orint STDERR iprmcid. "Unable co acquire lock on Stimes... writing an. ay.\n"; ) printf TIMES "Is %s\n\ Surl. Sdate; close (TUBES); tcleanup.cap: I ) elsif (Soperation eq 'List.All') ( ) elsif (Soperation eq 'History'I { locallScad) - '/usr/sbin/per Srlog2html Ssnapdir/Scurrent Surl Suser'; Send '- s/([\i%\SI)/\\Sl/g; printf STDERR "tslnvoxing %s\n*. iprmcid. Scad if Sdeoug; openlRLOC, "Sc dl") || iexicjraceiullyl'can't invoke Srlog2htal*. 0). while KR OGx) ( print S_; ) close(ROG); if IS?) ( if (Sdebug) ( print STDERR iprmcid. "Error S? from close(RLOGlVn* ; ) princ 'There was an error invoking rlog.'; ) else ( printf STDERR 'closeIROG) OFΛn" if Sdebug > 2; <λ KREF- s page.<br>\n!, Ssnapshoc. Surl Suser of the current page. (You will ze ac:.. tc ► • ""•.the list Suser eq "cniecbaages"; princ Strailer; ! icleanup.cap: I taken from htal-parse land reversed) SUD sucst tute.enticies ( local (Sstring) * (_; Sstring -- s/i/iamp'/g; Sstring =- s/</ilt;/g; Sstring -- s/>/4gt;/gr Sstring =- s/\'/iquot;/g; Sstring; ) SUD htmldiff.waiting ( print STDOUT * *. print STDERR 'Alarm...\n". SSIGI 'ALRM' )' = *htmldιf .waiting * . a.arτn(Scgι.ιeepalιvβ) , ) s.B byusercounc ( Scount(Sa) «=> ScountlSb), I sub Title ( local (St) = β_. return '<H >;L><TIT E>SC< I suo cleanup.dtp ( -f Sltmpf les == -1 i Sdebug) I print STDERR iprinti-: 'No temporary f.les t: clean .p 'n* / for I β trpfilesl -f ι-t S_) ( print STDERR iprintid '.nlinking S_ ' .s -1 S. if S ec.g -nlmi iS.) , I warn 'Can t unlink S_ S" else i print STDERR iprintid "S_ doesn't exist. 'n' .: idebug sub bydomam ( SdcmainxeylSai - idomaιnκey(Sa) unless defined (Sdssa nκey(Sal) , Sdcsaιnxey(Sb) = -domaince (SbI un ess defined (Sdomaιnκey(Sb)l ; return Sdomaιr.My(Sa) cap Sdoma nκey(Sb); ; sub doaainicey I locallSurll * »_, local (Sscnene.Saddress, port.Spath.Sguery.Sfragj = iwwwuri'parse(Surll ; t unless(Sport) ( Sport = 80. ) Saαdress -- s/ 1 )/\l\l/g I lower case - better way' wnere's my perl boe<" local Iβaddr) = splitIΛ /. Saddressl, return sprmtfC"is-//*ιs:%d%s". Sscheme. join!" " reverseIβaddr) ) , Sport. Spacbl. I sub byusercouncURL ( t reverse order of count, then normal order of URL within count. printf STDERR 'tslldl v Isltd π*. Sa, Scount(Sa), Sb, ScounC(Sb) if Sdebug > 3, localISres). f IScounc(Sa) '= Scounc(Sb)) ( Sres = ScountlSb) <=> Scount(Sa), printf STDERR tCount. SresVπ* if Sdebug > 3 ) else ( Sres ÷ ibydαmain; printf STDERR *\tURL. Sres\n * if Sdebug > 3; ) return Sres • locallSa.So = ιScount(Sa) Scount(Sb)) ) sub ur!2fr. i -seal(Surl) = »_. j.ocal!Sfr..ur.j = Surl. Sf.-.ur. =- s'-http://" • strip Sf.-.-r. =- s "—)>»'g » flatten printf STDERP 'Using file name \"ts\* for %s\n* Sfn.url Surl if Sαeoug > . ret'.m Sfr._.π ) sub rcsf.le ( localISfn) = β_. return *pages/RCS/Sfn.v * , • oca. ' moαe • End • • ' ■ti- IT, -• a comment to Keep perl from being confused •va. -xec per. -S S: il l*'!!' ] F-S.1 . _ ■■ . ' -.cr-e/αcuglis/ lib/perl" I , Seer-.: = : req-.ra • :;.-.;.. pi' re — := ' - •■ i-exit pi' Src:: = • -ic-e ccugus/ r-p/cgitest/ snapsnot* Soac<s-: = • -ere coup-is 'O n/nonandsBE* = seiectISTDERRi :.-._.: I:::- .ex::_cracefuli/("_rror n -παirtSroot; S" 2) -re.' i set all access cits on ?a.l:.i_.= :pe_s.:.:rs = Remesioe: Diff 'Status L st All 'Histcr' Users 5 eπe*r_jer if new All URLs' ' RL .sers Int .f ' i .-..-.Set' ' -?--;?arse . "input .: .ef r.eαlSir.p.ti ' tyFβ' l ) ] ( .'. lαefmeαlSmputl email ' ) ) I { Semaiimsg - " . else ' Sirs-put ( email ) = *S_S";r REMOTE .USER' I MSENVI 'REMOTE.HOST' 1 * Semailas. = ' Kι>ιnsert your usernaine before your nostname</ι> ) ' Scpticn.stnng = * * , ::r I laliowaole.operacions I ( Sopcion.scring = *<optιon> S. ' print .PnntJieader. print qq <title>S. HλNDS</tιtle> <n.><_-C λLI_N=ιnιddle SRC=' /lo_os/att_.llogo.gιf > Software and Systems ftesearch</h-> <hr> <boαy> <K_>NC -AlfDS</H-> <ιmg a..gn='rιghf alt=' |NO HANDS logol ' src=' /-douglιs/no_hands/logo.gιf> <h2>Us:.-.c This ?orm</h-> This fen is usea tc interact w th the <a href= ' -_ougιιs 'no_nands''>NO KAjπ)S</a> facility ϊou can remeacer what a pace pointed to oy a URL looks like, so you can return to it later _-.c see now it has changed I In Netscape, you can enter the QRL easily oy ridding cewn the right button over a l nk and selecting <ea>C:rγ trus ._•-* location to clιpboard</em Note that in Netscape 1 IN .f VOL dou-ie-clicc on the URL in the <em>locatιon<'em> line of tne pa;: /ou -ar.: zz tract, wnen you come back to this page and try to paste _:. _?_, -otr.inc w.il happen I You may view tne differences oet.es..-. :-e s: recent version you saw and the current version, or see :.-- -..szzr; z: past versions of the page (not only the ones you save. .... ;_u nay also see all URLs you have saveα away, or see information asc: ::-*r users cf the facility <P> Mote :_ι: __rr.r.:.y mere is no protection anyone can use any -T-... acαress anc can view eacn other s mroπ-atioπ <P> Scne c::'_rentat:c.". anout tne <a href=' /-dougi s/no_nands/ ">NO HANDS</a> fac:..-. Γ.C .-, ιs <a href="-douglιs/no_hands/help „__l'>foπn</a> in part;:.... are avanaoie NO HANDS works best in conjunction with an a_tC-_:.: <A Kϊϊ?= " -oouglιs/tracκ.urls/ *>notιt catιon syste_κ/λ> that tells vou tr.s: i page r.as cnanged and prc.-.i.i :r.e '.?_ :; access NO HANDS for tnat page <p:e> Oct 23, 1995^ /home/tball/douglis-tmp/cgi-bin/no_hands.cgi <p> ."_ <ιπp.: naι»e='url * type-*tex:" si2e-40> <r> ir... -ccr.ss <:r.p.: name-'eaa .' type-'tex:' vaiue = Sinputl 'email ) s _e-40> Semaiimsg <c> ;..:.:::-. <seιec: na*ne-'type * > < se.ec:> . < c:e> <-> <.".: :.-e= * s._πι:'> '.-p.: : pe-":eseC> < ::--> <-><.>.-:_: ::-_:r.c::on</ι><br> :r:-: .-ri-.tTrailer * -=s:?arse *.r.ρut ■ ::.--. ".:: e: irpieaentec <p>" -Prir.t.ariaolesllinput' »-.._Ξ; -. 'pecp-e' -« -w "pages' «. -d 'peop.e' .. -d 'pages* ' .: ;pe- TEST ">testf..e" i p:.-:: _.T αate printf TEST "T. k .ιs :es: succeeαe. - • .:cιi 'Sfoc :f;c - Erect =- s'.home/'/home. pr.-.:: 'qq S.ccess:.— , wrote <A „?_»= * file Is, testfιle">testfue</λ> T. 1 Sfco ..se pr.r.tf ■ '.π.ax e to cpen testfιle\r.' * ='-:_;:-:_:. llyι*c;:cest/peopιe or cgi/pa es not writaole directcry* 0). .: C6:ι-.ec Sinputl 'email') || c.:..-eα Sιnput('type') '! Sinputl 'email') eq ") ( * <j ; .:_;racen.llyCCSI script invoked without sufficient POST parameters, email-Siπputl email I cype-S nputCtype' I* 1) .rce: S.rere :cr I Siaiiowaoie. operations! ( S»rtere = S_ last if Sallowaoie.operationslS.) eq Sinputl 'type' I. .ex.:_-r.cef.ll/r Illegal operation SinputCtype ) specified". 1) _,-..££< cetmed Swhere. -.-.less Ξ.nput' type ! eq 'View urrent") ( pr.r: -Prir.tHeader .: S.-._.: r *type*) eq • Rer__mcer *v ! p:..-.:f *<T:tle>Reaιe__>er a U?L</Tιtle>\n' pr:*:: '<Hl>R--*__>er URL ts</Hl>\n", Sinputl 'url'), ) unless iSinputl'eaail') -- / * \»|\wt' ] • $/ j| SinputCtype') eq 'Users") ( -exι-_~racefull/l 'Script terminated due to abnoπsal input* . ■ email address must contain only alphanuaerics. <kbd </kbd> <lc-d>l</kbd>. or <kbd>'</kbd> and may not start * -.:- a o±c • f ,$._?.:( type ) eq 'Reoieaber' || SinputCtype I eq "Diff" || S:.-.p.:l type ) eq 'Status') f --..ess S.np-tC-ri'> =- '-http //|\w\ \l>-/\- '♦$?) ( .exit.-raeefuilyl'Scr pc teπunated αue to acnormal input' sp.ir.tf ''Is ( offending URL \'ts\' ) ' 'URL is restricted to <kbd>http //</k_>±> followed by alpanumberics or <kbd>/ I— </κ_>_>' S input ur il) -: :.τz-:' type . eq '.iff •• definedlSinp'.tl version ))) I :-<::: = i-,".p.t' version') .: =ex:r. -e 'peπj.t--ιate' ;« Sextra - ' d-\ \d«S/l ( --x.t.cracet.l./l'Versicr. :or diff specineα as Sextra r.o: acceptable" 1) :;-.- ST.."." • > .STDOUT*) :..".: : -€ =- s/ .5 ite: :;.:'.-_ S.r.putf type Sinputl ema.. Sinputl url i Sextra ::.-- • '::r.-->_rror prccessirc comrnand </strcr.g>M-* pr.-: . -.-:?:_..er • Local • cαe • _.-.d • /bm/sh -- « a comme-it to keep perl from oeing contuseα I Th.s scr.pt invokes πr__ld.£i or rcsdiff on multiple versions of a < file It can either run on a local file (fι.e=-user file) or a I snapsnot ot a file fne=http I These are treateα somewhat • differently • Act-all, use diff .rsteaα of rcsdiff and checx o.t cy hand to • keep the d.ff coαe .tself the same regardless eval exec peri -S SO S(1»'SI'» •f 0 Sdeoug = . Sneeo_.ocxs = . Ssnaps.-.ct = "http -v..126 researcn at: ccm/cgi-Di- -c.nanαs" unshift ιi::ιc '/home/co-glis ..b/perl'l • arcn.tect-re-spec.:.: .ibrar .for sys/socxet p- require arcn pi* Sarc- - tarcl if i Slialcc - ■ -cre/douς-.s arcn/Sarc- lib/per. ' .rsn.ttUINC S-.c.ϊci I • hat /ers on are c. -sing' i set this .t usinα S-.-e»er_dιrl Sl fc_Λ"._j-er = '0 40" unsmftlllNC SEN". I LIEJTΛV.PERL I • lome/douglis/ l.b/perl l._---v-perl-Slio_www_ver* I if (defined SENVl PAT" ) .. SEW I PATH ) ) ( I pr.nt STDEM 'Pat.- SDIVI PAW )<br> * .f Sdebug ) else { SENVl PATH ) : vhome/dougi s/arch/Sarch/bin /home/douglis/bin /bin/ /usr/sbin /usr/bm usr/ucb* ) uniess lSENVI httpjroxy ) ) ( • SENVl r.ojjroxy )='att coo' I want researcn act com to go outside SENVl πo_proxy l='tbu att com ncr com radish researcn att com wwwll2δ research att com ih att com ho att com cb att ecu rt att c:r SENVl rttp_proxy > = 'http / /radish research att com 8000/ ' require 'eg. -lib pi' SΣxrT_*"N = *cleanup_t_p' require "cgi-exit pi" require 'cgi-alirm pi' require "cgi -canon pi' require 'nornali.e-hcal pi" require "hostnaat pi ' require 'www pi' requ.re ' lock pi' .f Snβed_locxs require "pπntid pi' require 'cciae pi' select ISTDOUT) S | « 1 print -PrintHeader • arcr.tect.re-spec.:.: lιbrar tor sys/socket pi req-.re 'arch p.' Sarc- = larcn if -d Slibioc = ' icM / dougiis/arch/SarcVlio/perl' ] i nsn.ft llDiC S.inioci requ.re *.-_prcg p. ' Srcsp.f i Shcai_.fl Serf -.:.-_prog rrcsd.ff • " --__.d.ff 'diff «ex.:_.race:... ".nec.e to .::s:e rcsd.it r\tPAT- = 5_NV{ PAT" ' 0 ) if 'defined Srcsd.f f .exit.gracefully 1 ■ '.'r.anle to locate ht-nldiff 'PATH' 1 ' 01 f 'defined Shtmldiif. -ex:t_gracef .iy'".r.anle to locate d :: PATH')', D] f 'defined Sdiif Srcsdir = Sresdiff Srcsdir =- s'/rcsd iiS' SSUAPHOM- = SENV['_Kλ? ' I 1 " 'hone/αougiis, trp/cgitest/ snapshot' SSNAP = *SS:"APH.KE pages* •print STDERK "rcs-iii cgi eπtered v n" Ssnapsnot.autr.or = <<E3F <A HXEF=*ht:p . /ww * researcn at: cca/orgs ssr/peop.e/3ou;.:s/*>Fred _ouglιs</A> ECF cr.cp 1 S snacsr.c t_au _nc r Sht_ddιif_a'.tncr = <<ECF <λ HREF=*r.::p //wwv-∑pr in att ::n -tPall">Tcβ Ball<>A> ECF cio iShcnudiff.autr.cr Straner = *<Hπ"><F.*.T SIZE >T-.s page generated c, a "1 script written by Ssnapsnot. author " Sntmid ff.trailer = cq . using <ι><a .-ιrer = "http / 'ww.-spr .n att.com/-tball/ht_i_difi nt l'>htαιdιii< a>< ι> written c.. .-.:.-. .htnl = 0 .: ' 'tMethGet' ι • this or.e s done .itn POST -ReadPar'-i'i.-.p.: exit 1 :i lr.p-.t: 'd ff' ) ne d ff delete Sinputl diff'l. exit 1 .1 'defir.e.lSinputl f le'11 Sfile = Smpu:' iile'l. delete Sinput! file' ), if (defined Sinputl 'rβalf lβ'D I Srealfne = Sinputl'realfile'). if ISrealfi.e '- a'"$_K"_>/( [ \ \l\l\- MS'! ι printi "Invalid file specified Is * . Srealf le ) Sbase_fn = SI LOCK { if iSneed.locitsl ! print STDESB tprintii *Tryιng to lock Soase.fn\n' if Sdebug, openlLC . '*<SLOCK/Sbase.fn') || print STC-M Iprintid. 'Warning unable to open SLOCX/base.fn \r.'. last LOCK, tslocklLOCK. II II Pr 1 ""- STDEM .princ d "Waπung: unable to acquire lock on SLOCX/base.fn \n" last LOCr ) ) delete Sιr.putCreaUιle'1. Shtal = 1 Swww - 1. if (defined Sinputl'user')) ( Suser = Sinputl'user' ). delete Sinputl'user 1. ) else ' Suser = UNKNOHN. 1 ) else ' Swww = C Srealfile = .cc:_cancn!Sfιie! if iSfiϋ --- \ htmls/' Shu. = " . I .tap = • 'tap/rcsoiff cgi tl SS" * __p/rcsdiff cc. :2 SS"). Soutput = ' _rς: rcsdiif cg: ciiϊ SS' ltmpfi.es =(«t=p Soutputi -cleanup. tr-p Sversions = . Sverstr = •■ local l*vers crs :cr $J 'sor: z - .rsicn *.evs unpuc pushιβvert.:-E S ct 23, 1995^ /home/tball/douglis-tmp/cgi-bin/rcsdiff.cgi i .f (Stversios = ll ( pr.-tf ''Must specify exactly two versions tc d.fi Got join eversionsl) Ifiles = I) fcr Sv leversionsl I .f 'Shtal 'I Swwwl ( I checx out temporarily .f (Sv eq 'current*) ( if (Swww) ( Sanvepid = -cgi.keepalive print STDERR iprintid 'Keepai * e p.d Sai.vep.c .f Soebug Songurl = Sfile Sresponse = -www lrequestl GET "...e '-eaoers "ccntent i kill : Salivepiα I SIGHT .£ I 'wa tpiαlSalivep d 0>J { prirtf STDERR 'Is warr.irg _-__ue to ι.a.: -cr c ic keepai. Je process Salivepid v ipr.-t.d I .£ ( Sresponse '- /"2/ I ( local (Srest) = Swwwerrcr Resp' , sssaςe l Sresp.rseι • .f (def.nedlSrest) l ( Srest = ' Srest" I .exιt_gracefullyl'Response Sresponse cur.ro GET requests s: ■ 1) ) print STDERR iprintid 'Noππaliz.r.g URL Sfile -" .£ Sdebu, Sconcent = tnor aliieht lISconcent Sfilei .f IScontent -- /"Error filteπn; HTKL /I ι iexit_.race-ul-ylScon.enc. 0) 1 Sttrp * pop(Itap). openlTMP * >Stιn>"l 11 .exit.gracefully!"Can t open Stop' 0) print Vf? Scontent, closelTW) push (Ifiles StropI , ) else { push (Ifiles Srealfile) ) ) else ( S__p : popllQBP) system r Sco -pSv Srealfile > Strp' l tt texic__Ta.efullyf.iror running Sco -pSv Srealfile > Stop S " 0) push lifiles St-pl ) ) else ( Svβrscr = '-rSv ' if Sv ne "current' ) ) .! l -s SfileslO] == -s Sfιles( l) tt s/stemc bin/cap' Sfiles lO] , Sfiles l ll l ) ' print * <h2>No differences encountered</h_><br>\n* ) eisif (Shti l I Swwwl ( Sf ilelist = .oinl βfilesl -f IS-tnl) ( Strailer -- s/ \ S 'Sht ld ff.trailer / openlD.™ 'Shtmlc.fi Sfilelist Soutput 2>.l I " M lexιt-_race.ully( 'Can t invoice Shtiruα.ff S 1 " 0) wruιe(<DIFF>) ( printf STDEHF S_ if Sdebug cose (DIFF) pr.r.tf STDERR 'Errcr S' from closelDI" \"' .'. S' .- Sdebug openlDIFF "<Soucpuf) || lexit_grace:...y("Can t open temporary output" 0) i else ( open(D: r F "Sdiff Sfιlelιst|") || .exit.gracefullyI'Can t invoke Sd.fi S ' 0 Shtal iconte"- = joinl ' -' <DIFF> ) • r " cioselDIFF', :: I 'SwwwI ■ iocallShαsci = i.-.oscname FQDN. « Surl = • htcp./.'Shcst/Sfiie' Frtnt inomaiite.-.tal (Sconcent. Surl I ) eise ! print Sconcent. print S railer i: ISww i • iocaitSreme-Oer = cjq'<l > <A HJ",EF=*!.sr.apsr.oc 1 e- ; _i = -£sriurl=Scr:sirιttype=Re emDer'>Reιrιeιrιber current version": /λ> <cr> -f , Suser eq 'tr-ecπanges' ' • specia. case :cr r.o-. Sremei-Der = ** '.;> <A H?EF='Sorι;_r:'>Yiew ■ -rrent versιcn< A> Sremenoer <<ui> E3F ) else ' print " <K"" L><T..__>-i i I -c S£ιle< 71""__> n<BCBϊ> ■ -<??._> - • open.F.CS-IFF *Sdι:: -c S erstr Srealfile ', i exit 1 Slιnes=C -rule ι<R___irF>l I αe-πtmlify : wnat e.se is r-issir.α' print Sl nes*- print *<h2>No differences if fSlines. print • </P E></BCDΪ>Straιier</KT*_.>\n' ) ) else ( print "<HT!_.><TI'π_;>rcsdιfi -c Sverstr Siιle</?I""_-><n<B--Y> v r.<?„_>\n * openlRCSCI:-. "Sdii! --. Sverstr SrealfileI" '| exit 1 Slιnes=C. wniie (< CSDIFF>ι ! • de-ntaiify it .nat eise is missing 1 print. Slines**. 1 print '<h2>No differences enco_ncered</h2>\r." if CSlmesi. print • </p?_-></BODY>Stra ιer</!Tn_.>\n'. ! icleanup.tnp: cleanup.tap l if (Sltmpfiles == -1 ii Sdeougl ( print S -E?- 5 iprintid 'No tesrpcrary files to clean up v.' for I ._mptil.es ' ' if l-e S. • print ST-BUI tprir.-.i. "Jnimxing S. ". Is -1 S. :i See:.; unlinxlS. ,' warr. 'Can't uniinx S_ S" pri.-.t STDERR ipnr.c:. '£_ dcesn't exist . .-' .f Saeoug sub oyversion ( return -1 if S eς 'currer.:' return 1 if a e*; 'currer.:' 5a <=> Sb ■ cir. sn -- 1 _ cσπnenc to keep perl from Deing confused .... exec per. -_ S3 S I 1*"SI" ! p.s- <::;: • -cme/dc-clis/ lib/perl " ) . ire.: ; ' .-.cne dcugiis trφ/cgit-st, snapshot' sta:<_— = ' --re/cc-c.is/oin/r.cr.andsTest* • new versicr. -ii-. ' Scl-ir. = se.ect (STDERR ) s . β c 'STX s ι = ; . ::.c.r Sr::: .ex.:. race:ul .. ' 'Error in r dir'.Srcc:' S " _-.*< . 1 set all access cits on •ϊ..:.i_.e cperaticπs = Remerrxer . 'Diff 'Status List All Histcrv Users Remember if new All URLs' "RL users Vie. :.rr.^ Ir: .! iXethGet k?«adParse'"ιnput! .: . defιnedlSιr.p'.t('type'l!) I : i '.defined!Sinputl 'email' 111 ( Semailasg * " ,< else ( Sinputl 'email 1 • 'SEI I'RBIOTE.US-ll' )\ISENV( 'REMOTE.HOST' I ' Semaiit-sg * *κι>ιnsert your use name before your hostname</i>i * . Scpticr.-Stπng * ". for (tallowaole.operationsl ( Soptior..string « "<optιon> S. ": print tPiiπ-Hβader, print c-ς <ht"".i> ctιtle>NO HANDS*/t tle> <h2><: • *_ ALIGN*mιddle SRC**/logos/attbllogo.gιf* > Soft.are and Systeαs Research*/h2> <hr> <body> < mg ai gn * 'rιghf ait * ' (N0 HANDS logo) ' src«' /-douglιs/no.h_nds/logo.gιf ■ > <tι2>.sι.-.g This Foπr /h2> This fsro is used to interact with the <a .iref** '-dougiis/no.nands/ 'XtO KAHDS</a> facility , You can reαaiaber ■ -nat a page pcintec to by a UKL looks like, so you can return to it later and see now it has changed. (In Netscape, you can enter the URL easily cv ncldir.g down the right button over a link and selecting <em>Copy this inx iocation to clιp_oerd</-m> . Note that in Netscape 1 N if γcu doucle-click on the URL in the <eα>_ocatιon</eα> line of tne pace you want tc track, when you come back to this page and cry to paste :-e ' R_ r.ctr.ing will happen I You may view the differences cet.e.r. :.-.e aosc recent version you saw and the current version, or see :.:e .-.istcry c: past versions of the page (not only tne ones you ..-. • ee a-ayj :;_ mav also see all URLs you nave saved away or see mfcrcation accu: other users oi the facility <P> .'lets :. * .Ϊ: :_rre.-.:.v there is no protection: anyone can use any _:_... aocress and can view each otner's information. Sore _c._menta:ιcr. aoout the <a href * * /-douglis/no_hands/ *>NO HANDS</a> f-ci.i-v anc te.is <a nref * ' /-douglιs/no_hands/help .n_nl*>for-ι</a> ir. parti:.. ar are available NO HANDS works best m conjunction witn ar. aut:-itιc <A HF.Er * " -douglιs/track_urls/'>nocιfιcaticn syscem</λ> that te.is you that a page has changed and crevices :ne 'JRL tc access NO HANDS for that page <i:— r_:.-.;d=p_sτ 'r> '.-_, <ιr.put naιne« *-.rl' type«'text* sιze«40> <:> Ir:.. __o:ess ^inp t narwemail * type * 'texf va.ue = Sinp-ti er-.- sιιe*40> s__aιit_sg <r> Operatitr. <seιect name*' type '> Soptιcr..s;r:r.g <.seιect> < r pre> input type * "submit • > <input t',-pe * 'reset • > <'fαr-ι> P Sot. that t.-.is page is the <o>deveiopment</b> version oi the facility Ii ye..-.ave :r:_cie with it you may find a more <a href * '/cgι-bιn/no_hands"> s:_c e versicn</A>. < --<Pχ;>'r._er construction*: /ιxbr>--> prir.: iPrinrTraiier. e.se ifieacPar** ■ " input S.x::a = " • pr:" • *;:: .e: ir-piemented <ρ>'. iPrintVariaolesllinput: --..ess -. 'pecple * • * -w 'pages' ϋ -d "people" •• -d 'pages' i ! • ..' cper.iTEST ">testfιle"] I I ■ printf TEST date 1 printf TEST 'This test succeeded. 1 .cca iSfooi ■ Sfoo = Sreoci *- S'/home/'/homel/'. ■ prir.:: '^ Successfully wrote <A HRE- 'fιle:»s/testfile'>testfιle</A> .-. . sf:: . " i else ( • printf "r.able to open testfile\n*. ■ ) fcexit.gracefully' 'cgitest/pβαpie or cgi/pages not writable directory*. 0). if defined Sinputl 'email'] : defined Sinputl 'type') j, Sinputl 'email') eq "1 ( iexιt___acef'_ll l'-_I script invoxed without sufficient POST parameters, email «S input ' email'), type * Sιnputl 'type' )' . C uncef Swhere fcr s; Stallowable.operationsi I Swhere = 3_. last if Sallowable.operationslS.,' eq SinputCtype'!: iexit.gracefullyc Illegal operation Sιnρut('type') specified", 11 uniess defined Swhere: unless (SinputCtype ) eq "View Current" I ! print iPr.n_Kea.er. if '.Sinputl "type" ! eq ' einβ-iDer ' ' print: "cTι:lβ>Rβι-β__>er a URL</Title>\n* : printf -<Hl>Re-iemoer URL: ts</Kl>\n*. Sinputl 'url' ) : uniess ISinputl'email" 1 »- /*\w| \w»! . | •$/ || Sinputl 'type* I eq 'Users") ( -ex t__racefully! 'Script terminated due to abnormal input*. 'email address must contain only alphanumerics. <kbd>.</kbd>. <kbd>><fkbd>. or <kbd>'</kbd> and may not star: wit.n a <xsd>> if (SinputCtype ) eq 'Remember* '| SinputCtype') eq "Diff || Sinputl type ) eq "Status* I ( uniess (Sinputl 'url') * - .'"http:// |\w\ .\l!-/\-: )*S?) ( •exit.gracefully! 'Script terminated due to abnormal input*. spπntf '*ts loffending URL: \*%s\*)*, 'URL is restricted to <kbd>http /<7kbd f.liowed by alpanumberics or <kbd>/.*--b' </x_<->*. Sinputl "url")) I. ..' : S input C type ; eq 'Cif!' i. definedlSinputl 'version' )) ) ( Sextra s Siπp :' version': :i 'Sextra r.e "penultimate" ϋ Sextra '- ,'"\d * \ . \d*$/ ) ' iexit_:racef_lly( 'Version for diff specified as Sextra nc: acceptable' '. Sdebug - open I ST-EEP. ■ > i-TDO-T' ) . pπnt!l"<PRE>ιr.": . J incut!' type i •- a /_/.. exe: Sbacxer.c. i.-.pu:! 'type pr:r: * <-:r:-g>_rr:r processing Oct23 ) 1995 V^ /home/tball/douglis-tmp/cgi-bin/snapshot.cgi ϊl Local ariables mooe perl ' End '

Inventors: BALL THOMAS J, DOUGLIS FREDERICK, BALL, THOMAS, J., DOUGLIS, FREDERICK
Applicants: AT & T CORP, AT & T CORP.
Application year: 1996
Main classification: G06F17
IPC codes: G06F9, G06F12, G06F17