From: tsarver@andersen.uucp (Tom Sarver)
Subject: Re: Disk Directory Manager
Summary: Multi-volume file system (LONG)
Organization: A Consulting Company
Date: Wed, 8 Apr 1992 20:58:12 GMT

In article <29E1057E.7049@ics.uci.edu> dkim@ics.uci.edu (D.Kim) writes:
>I need to know if there is an utility for managing files and disks?  I
>only know of DirMaster which keeps track of directory of all your floppy,
>but it is too limited and too slow.
> [Deleted story about how much he needs this.]
>If you have any inputs or suggestions, please let me know.
>
>Here are the features I want (and may not be possible in Amiga OS):
>-virtual disk (VAD:) which acts as 1 big harddrive composed of floppy
>	disks.  (about 500 DSDD disk equals 500 meg )

Awsome!

>-automatic compression/decompression (68020/881 and above) to double
>	DSDD capacity to 2 meg.R~?
>-HighDensity drive, other non-HardDrive scsi storage devices support

Any storage device, and when the VAD: needs to expand, it should give user
question about device (eg, HDfloppy, RegFloppy, Syquest, etc.)
What about Tape?  Too clumsy?

>-directories are stored in Hard Drive for speed and efficiency (simply
>	they would be link to the physical files in the disks.)

We're talking about a HUGE file to store the directory to 500 disks.

>-VAD: should be completely transparent to CLI, WB, Applications...
>	and completely compatible.
>-Allow a file to be bigger than 880k (split into many disks) and be
>	completely transparent.
>-Automate everything.  It'll ask the user to enter a blank unformatted disk
>	if more size is needed by VAD:.  User only needs to write the disk number
>	on the disk.

See above comment.

>-minimize disk access.  Since directory is located in Hard Drive or Ram,
>	browsing through directories will not access floppy.  Only when the
>	actual file is needed would the request for disk #x appear.
>
Here's some more:
When copying to VAD:, you should be able to place the POINTER to the file in
as many locations in the "directory" structure as you wish.  This is similar
to the Unix file system with logical links.  This feature is useful for having
multiple ways of organizing one's files.

>I am a novice when it comes to disk devices.  So if you think it can be done
>by implementing a handler (L:VAD-handler) and mounting it, then let me know.
>
Yep, you'll need a handler if you want it to be accessible from the system
transparently.
Here's how to get started:
Consider any disk file system.  Barring a few design decisions, this is basicly
how it goes:
You have a set of directory entries (at least one for each file AND directory)
and a pool of space (disk blocks, usually) in which to put files.  The directory
entries hold information about the file like the filename, its creation date,
last change date, length, type, owner, protection bits, etc.  It also has some
pointer(s) into the pool (eg, "Block 34 to 36" and "Block 128 to 190").
A file system with with symbol links have a sort of half-structure which holds
some information (like filename, owner, etc.) but is primarily a pointer to
the "real" directory entry.

This is the general structure of a file system, give or take organizing the
directory entries and what a pointer into the pool looks like.  I also haven't
touched on error recovery (which differs based on other decisions made).

When a file is deleted, the directory entry is marked as deleted (or actually
deleted) and the space in the pool marked as available (in some cases it is
added to the special directory entry which keeps up with free space).

When a file is inserted, a directory entry is created (or filled) and space in
the pool is allocated, and the pointer(s) to this space is placed in the dir.
entry.

When a file grows, may simply grab the block immediately following the current
last block.  If the next block is occupied, the system would have to create a
new pointer to a new area of the pool.

Detour:  Disk compactors do their thing by reducing the number of pointers
from a dir. entry into the pool, which is accomplished by moving stuff around
in the pool.

OKAY!
Now, a multi-volume file system would (as your suggestion) keep its directory
entries in a file on the hard drive.  This means we can easily keep variable
length directory entries (ie, any number of pointers to the pool).  The pool
is going to be the 700 floppies (or other removable media).

A directory entry will contain some subset of the information listed above
plus some number of pointers into the pool.  What does a pointer look like?
I suggest treating each floppy as a contiguous set of disk blocks.  Therefore,
a pointer would be a volume number, a beginning block, and ending block.

If this is all the information we had, it could become a full file system.
However, if we ever lost our directory structure, we would have no hope of
rebuilding our file system.  There are a few ways of dealing with this.
One way is to maintain a block header which is reserved for a pointer to the
next block in the file (and a special value for end-of-file).  If catastrophe
struck, we would get all our files back, but we would have no idea about
file names or where each lies in the hierarchy.

One alternative is to back up the directory structure.  If disaster struck
we would have no idea about files which were added since the last backup,
AND we might "lose" some existing files if they were deleted and then the
space used by another file.

In addition, you will need some utilies for manipulating the file system.
For example, you would probably like a compactor, which would reclaim space
which is too small for the average file.  (One would want to trade off between
the number of swaps and the amount of saved space.  Something linear to the
number of disks wouldn't be bad.)

One would also want the ability to move some of the pool to new media.  For
example, you may have just bought a new High Density floppy or, better yet,
a Syquest or floptical drive.

One would also want a utility for recovering files which were on a floppy
that went bad.  At least copy them on to another floppy (the files which are
still around).

FINALLY!
Caveat Emptor:
This could all be a lot easier if this were a Write Once Read Many (WORM)
device.  One wouldn't have to bother with expanding files or recovering
space from deleted files.  But one would still need some recovery mechanism.

>INET:  dkim@bonnie.ics.uci.edu
>

It's been fun!
--Tom Sarver				 tsarver@andersen.com, Chicago, IL
|> I don't actually live in the physical world;  I just have to visit often<|
|>                        enough to go to the bathroom.			   <|
The opinions presented here are not necessarily those of my employer.
Newsgroups: comp.sys.amiga.applications