# Presentation: 125335

## JLab Lattice Portal – Data Grid Web Service

- Ying Chen,  Chip Watson
*Thomas Jefferson National Accelerator Facility*

## Jlab Lattice Portal

- Jefferson Lab (JLab) and its collaborators at MIT are in the process of developing a web portal for the Lattice Hadron Physics Collaboration.
- This portal will eventually provide access to Linux clusters, disk caches, and tertiary storage located at JLab, MIT, and other universities.

## Jlab Lattice Portal

- Jlab Lattice Web Portal infrastructure capabilities:
- 	- Publish/locate a data set.
- 	- Submit/monitor/control a batch job.
- 	- Move a data set to/from the compute resource.
- 	- Authenticate/authorize and track the use of the resources.
- 	- ...

## A three tier web services architecture

- Web Browser

- XML to HTML servlet

- Web Service

- Application

- Web Service

- Web Service

- Grid Service

- Local Backend Services
- (batch, file, etc.)

- Web Server (Portal)

- Authenticated connections

- Remote Web Server

- Web Service

- File Daemon

- Grid resources, e.g. Condor

- Batch Daemon

## Why Grid Web Service ?

- Disk Mgr

- Tape Mgr

- users

- Managed disk

- Unmanaged disk

- tertiary storage

- users

- users

- login

- login

- login

- Data Grid   Node

*Fire wall*

## Data Grid Web Service

- Hides the complexity of the underlying system
- 	*users don’t have to deal with the disk manager or tape manager etc.*
- Provides the service on the web
- 	*users don’t have to login to do the work*

## System Architecture Overview

- Grid Service

- Disk Mgr

- Tape Mgr

- users

- Managed disk

- Unmanaged disk

- tertiary storage

- users

- users

- Web browser

-    web browser

*Fire wall*

## Web Based Grid Service

- A web based grid service is implemented as a java servlet. It performs four basic tasks.
- 	- gets the request from user
- 	- checks the authentication for a privileged service
- 	- processes the request by invoking the grid server
- 	- sends the results (responses) back to the user

## import javax.servlet.*;
import javax.servlet.http.*;
import java.io.*;
public class GridService extends HttpServlet {
	public static String url2 = “https://xx.xx.xxx/servlet/GridService”;
	private GridServer service;
	public void init() throws ServletException  { service = new GridServer(); } 
    	public void doGet (HttpServletRequest request, 
                          HttpServletResponse response) throws ServletException  {
         response.setContentType("text/xml");
         String req = request.getParameter("service");                      // get request
	    if (req == null ) req = “<list name=“foo”/>”;
	    try {
	    	if (!request.isSecure())  response.sendRedirect (url2);   //redirect to https
  		else {          service.process(req);                                   // process request
         	    service.printResult(response.getWriter());   }              // return result
	     } catch (IOException e) { e.printStackTrace(); }
     	}
}

- import javax.servlet.http.*;
- import java.io.*;
- public class GridService extends HttpServlet {
- 	public static String url2 = “https://xx.xx.xxx/servlet/GridService”;
- 	private GridServer service;
- 	public void init() throws ServletException  { service = new GridServer(); }
-     	public void doGet (HttpServletRequest request,
-                           HttpServletResponse response) throws ServletException  {
-          response.setContentType("text/xml");
-          String req = request.getParameter("service");                      // get request
- 	    if (req == null ) req = “<list name=“foo”/>”;
- 	    try {
- 	    	if (!request.isSecure())  response.sendRedirect (url2);   //redirect to https
-   		else {          service.process(req);                                   // process request
-          	    service.printResult(response.getWriter());   }              // return result
- 	     } catch (IOException e) { e.printStackTrace(); }
-      	}
- }

## public class GridServer {

-     public static String myName =
- 	"<grid name='demo‘ url=‘https://xx.xx.xxx/servlet/GridService’>";
-     private String request;
-     public GridServer () { }
-     public void process (String req) {   request = req;  }
-     public void printResult(PrintWriter out) {
-          out.println(myName);
-          out.println("\t"+req);                        // echo the request
-          out.println("\t<success/>");             // send the result
-          out.println("</grid>");
-     }
- }

## Web Based Grid Service

- Jlab Data Grid Web Service - a java servlet
- 	- uses apache with model-ssl as secure web server and tomcat as servlet engine
- 	- service request is an xml format string
- 	- the result/response is also in xml format
-  	- a certificate is required for the privileged services

## Service Request/Response

- Request:**   **
- 	<info name="version“ />
- Response:
- <grid name="LQCD at Jlab“ url="lqcd.jlab.org/servlet/grid.data.Service">
- 	<info name="version">
- 	    <info>jlab data grid version '0.1'</info>
-       </info>
- </grid>
-                  ---------------------- or -------------------
- <grid name=“LQCD at Jlab” url=“lqcd.jlab.org/servlet/grid.data.Service”>
- 	<info nam=“version”>
- 	     <error>Attribute “name” missing</error>
- 	</info>
- </grid>

## Why xml ?

- Self-describe (easy to understand)
- Language independent
- easy to assemble from multiple source
- Interchange of data among application
- Focus upon the data structure *(can easily convert to different presentation)*
- Many free or commercial tools available to process xml data

## http://lqcd.jlab.org/pbs/

- _[http://](http://lqcd.jlab.org/pbs/)__[lqcd](http://lqcd.jlab.org/pbs/)__[.](http://lqcd.jlab.org/pbs/)__[jlab](http://lqcd.jlab.org/pbs/)__[.org/](http://lqcd.jlab.org/pbs/)__[pbs](http://lqcd.jlab.org/pbs/)__[/](http://lqcd.jlab.org/pbs/)_

## Jlab Web Based Grid Service

- The grid service includes:
-    - provides node information and status
- 	- lists the contents of the grid node
-    - moving files within the node
- 	- performs file operations *(delete, pin/unpin, mkdir, mkdirs etc.)*
-  The URL of the service: *http://lqcd.jlab.org/servlet/grid.data.GridService*

## Slide 16

## List Service

- Lists the contents of a given file system.
*<list name=“$CACHE/ychen” level=“2” />*
**Name** attribute (required) specifies the file system (directory)
**Level** attribute (optional) specifies depth of the directory to list.
- The contents of the directory in xml trees will be returned if a valid directory is requested.

## Copy Service

- Copy the source file to the destination.
- 	*<copy source=“$DATA/foo”  destination=“$CACHE” />*
- The required **source** and **destination** attributes specify the source and destination location on the node.
- Two Boolean attributes **overwrite** and **permanent** are optional *(default value is false). *
- The error message (as text of the error element) will be returned to the user if a copy fails.

## File-operation Service

- Deletes a file or directory, creates a directory or directories, pins or unpins a file, etc..
- 	*<file-operation type=“delete” name=“$DATA/foo” />*
- Required attribute **type** specifies the type of the operation (*delete, mkdir, mkdirs, pin, unpin and permanent) .*
- Attribute **name** (required) identifies the file or directory on which the operation will be performed.

## Use the Service - directly

- Enter the url in a web browser:
- _[http://](http://lqcd.jlab.org/servlet/grid.data.GridService?service=%3Clist+name=%22$CACHE/home/ychen%22/%3E)__[lqcd](http://lqcd.jlab.org/servlet/grid.data.GridService?service=%3Clist+name=%22$CACHE/home/ychen%22/%3E)__[.](http://lqcd.jlab.org/servlet/grid.data.GridService?service=%3Clist+name=%22$CACHE/home/ychen%22/%3E)__[jlab](http://lqcd.jlab.org/servlet/grid.data.GridService?service=%3Clist+name=%22$CACHE/home/ychen%22/%3E)__[.org/servlet/grid.data.](http://lqcd.jlab.org/servlet/grid.data.GridService?service=%3Clist+name=%22$CACHE/home/ychen%22/%3E)__[GridService](http://lqcd.jlab.org/servlet/grid.data.GridService?service=%3Clist+name=%22$CACHE/home/ychen%22/%3E)__[?service=<list+name='$CACHE/](http://lqcd.jlab.org/servlet/grid.data.GridService?service=%3Clist+name=%22$CACHE/home/ychen%22/%3E)__[ychen'](http://lqcd.jlab.org/servlet/grid.data.GridService?service=%3Clist+name=%22$CACHE/home/ychen%22/%3E)__[/> ](http://lqcd.jlab.org/servlet/grid.data.GridService?service=%3Clist+name=%22$CACHE/home/ychen%22/%3E)_
*                               ***or**
- _[https://](http://lqcd.jlab.org/servlet/grid.data.GridService?service=%3Cfile-operation+type=%22delete%22+name=%22$CACHE/home/ychen/foo%22/%3E)__[lqcd](http://lqcd.jlab.org/servlet/grid.data.GridService?service=%3Cfile-operation+type=%22delete%22+name=%22$CACHE/home/ychen/foo%22/%3E)__[.](http://lqcd.jlab.org/servlet/grid.data.GridService?service=%3Cfile-operation+type=%22delete%22+name=%22$CACHE/home/ychen/foo%22/%3E)__[jlab](http://lqcd.jlab.org/servlet/grid.data.GridService?service=%3Cfile-operation+type=%22delete%22+name=%22$CACHE/home/ychen/foo%22/%3E)__[.org/servlet/grid.data.](http://lqcd.jlab.org/servlet/grid.data.GridService?service=%3Cfile-operation+type=%22delete%22+name=%22$CACHE/home/ychen/foo%22/%3E)__[GridService](http://lqcd.jlab.org/servlet/grid.data.GridService?service=%3Cfile-operation+type=%22delete%22+name=%22$CACHE/home/ychen/foo%22/%3E)__[?service=<file-operation+type=‘delete’+ name=‘$DATA/](http://lqcd.jlab.org/servlet/grid.data.GridService?service=%3Cfile-operation+type=%22delete%22+name=%22$CACHE/home/ychen/foo%22/%3E)__[foo'](http://lqcd.jlab.org/servlet/grid.data.GridService?service=%3Cfile-operation+type=%22delete%22+name=%22$CACHE/home/ychen/foo%22/%3E)__[/> ](http://lqcd.jlab.org/servlet/grid.data.GridService?service=%3Cfile-operation+type=%22delete%22+name=%22$CACHE/home/ychen/foo%22/%3E)_

## Use the Service - Grid API

- A grid in_[terface](file:///docs/index.html)__[ API ](file:///docs/index.html)_is designed for using this Data Grid web service.
- A java implementation of this API is being developed to encapsulate this xml service.
- Using this API to obtain the service, the xml input and output are hidden by the implementation.

## Grid API - GridFactory

- public class GridFactory– provides an interface to access the GridNode object using static methods.
-  	 	*public static* *GridNode getGridNode (String url)*
* **	**	**public static* *GridNode getGridNode ()*
*	**	**public static* *GridNode getReplica ()*
- interface GridNode –* Is **a grid node or replica catalog object that can be obtained from the GridFactory class.*

## Grid API - GridNode

- public interface GridNode {
- 	public String getURL ();
- 	public String getName();
- 	public GridDirectory getDirectory (String path);
- 	public GridFile[] list(String path);
- 	public String[] listNames (String path);
- 	public boolean isExist (String path);
- 	public boolean isDirectory (String path);
-  	public boolean isFile (String path);
- }

## Grid API - File/Directory

- GridFile

- GridLink

- GridLogicalFile

- NodeDirectory

- NodeFile

- ReplicaFile

- ReplicaDirectory

- GridDirectory

## Use the Grid API

- public class GridAPIDemo {
-    public static void main (String[] args) {
-       try {
-            	GridNode jlab = GridFactory.getGridNode();
-            	 NodeDirectory dir = (NodeDirectory) jlab.getDirectory(args[0]);
-            	String[] names = dir.listNames();
- 		System.out.println(args[0]+“ directory contains:”);
- 		for (int i=0; i<names.length; i++)
- 		    System.out.println(names[i]);
- 	} catch (Exception e) { System.out.println(e.getMessage()); }
-     }
- }

## Use the Grid API

- An applet using the grid API is being developed to manage the files within a data grid node.
- 	_[https://](https://einstein.jlab.org/grid/FileManager.jsp)__[lqcd](https://einstein.jlab.org/grid/FileManager.jsp)__[.](https://einstein.jlab.org/grid/FileManager.jsp)__[jlab](https://einstein.jlab.org/grid/FileManager.jsp)__[.org/grid/](https://einstein.jlab.org/grid/FileManager.jsp)__[FileManager](https://einstein.jlab.org/grid/FileManager.jsp)__[.](https://einstein.jlab.org/grid/FileManager.jsp)__[jsp](https://einstein.jlab.org/grid/FileManager.jsp)_
- User can manage their files via the web by using this tool.
- With other web services in the Lattice Portal (batch system web service, monitoring, etc.), most tasks can be done without logging in.

## Future Developments

- Standardize the xml request and response.
- Add additional web technology (SOUP, WSDL, and UDDI) to build dynamic web-based system
- Extend the batch web interface/service.
- Add third party file transfer capability using remote web services (web server to web server)

## Questions ?