From: weave@apache.dtcc.edu (Ken Weaverling)
Newsgroups: alt.hackers
Subject: Gotta hack into DOS's COMMAND.COM :-(
Date: 22 Jan 1993 09:20:36 -0500
Organization: Delaware Technical & Community College
Distribution: world
NNTP-Posting-Host: apache.dtcc.edu
Summary: I hate MS/DOS

Yuck!  Looks like I have to lower myself and hack into DOS's COMMAND.COM
program.  What a drag...

The problem I have been having is that my users on our PC network have been
having Sharing Violations triggered and the infamous "Abort, Retry, Fail"
error to greet them.

Naturally, they freak out and choose the wrong one (hardly ever Retry which
almost always works since someone else had the file lock). 

However, once in a while, someone must reboot or something and a batch file
or EXE file gets stuck in locked mode, and no one else can run it.

A real bummer from an admins standpoint.

We run Locus's PC-Interface which is nice because it is bare-bones
networking, doesn't chew up RAM space, and allows our Unix hosts to 
function as DOS file servers and I can do all administrative duties from
Unix 

(Indeed, we also have software that turns the same Unix box into a Mac
Appleshare compliant server so user files are always accessible whether they
use DOS, Unix, or Macs... but I digress...)

So I contacted Locus and after several mail messages, the poor front-end
non-technical person helping me drops a note on me which I should have
realised way before hand myself.   But it leads me to the conclusion that
I have to start hacking at DOS!  

Below is my latest mail to Locus.  Anyone else out there have any clues?

============= cut here ============

> So, more specific questions are...
> 
> 1) What causes a DOS EXE file to get locked? Does COMMAND.COM do this?
>    Or does certain utility programs lock themselves when they run?
> 
>      'TIS THE WAY THE FILE IS OPENED. IF A FILE IS OPENED IN WRITE DENY MODE
>      NO OTHER USER CAN OPEN THE FILE UNTIL THE FIRST USER CLOSES THE
>      FILE.

Thanks, this tells me a lot!

The DOS technical reference manual says...

"If the file was successfully opened on one of the other sharing modes
 and an attempt is made to open the file again in Compatibility Mode,
 an interrupt 24H is generated to signal this error. The base interrupt
 24H error indicates Drive not ready, and the extended error indicates
 Sharing Violation"

Now, INT 24H is the "Critical Error" handler and is what puts up the
ole Retry, Ignore, Fail message, so this looks like what is happening.

However, since it seems like DOS's COMMAND.COM program is the guilty
one in all of these cases, it is probably safe to assume he opens 
files in the same mode every time.  ie, since I have had locked EXE
files and BATch files, it must be COMMAND.COM that is opening them.

But if he always opens them with Deny Write (like he should), then that
should allow others to open the file for Deny Write access. 

But the next paragraph in the DOS tech manual offers a clue...

"The sharing modes for a file opened in compatibility mode are *CHANGED*
 by DOS depending on the read-only attribute of the file. This is to
 allow sharing of read-only files"

Ah, what the heck is going on here?

>From the open table in the manual it indicates that a read-only file opened in
compatibility mode, read-only  will implicitly cause a file to be opened in 
"Deny Write" mode, like you said.  

Ah, which brings us around to the top of my hypothesis above. If a file
is opened in Write Deny mode and someone else attempts to open it in
compatibility mode, it triggers the critical error!!!!!

So, if the above theory is correct, then I'd say the *bug* is Microsoft's
since a lot of programs don't specify sharing modes and simply open 
files in "compatibility mode" and a read-only file should logically
never be denied access for read-only operation.

So, the possible solutions are...

1) Make the DOS executables read/write access. A real bummer. This would
   open up possibilities of students copying trojans or other nasty
   stuff to publically accessible files

2) Get Microsoft to fix it (yeah, right!)

3) Have Locus alter the default behavour. I'd say that this would be
   dangerous and not a good idea, since it won't work as documented by  
   Microsoft and could break a lot of stuff.

4) Request Locus to allow users to remove record locks from Unix without
   having to log out all PCI users first.  (Indeed, I could do this 
   with sharectl -r under server version 2.8.7 but if I do this with
   4.0, everyone's login hangs at the next file access...)  Ideally,
   a Unix utility program to list record locks and allow individual
   deletion of them would be the most ideal.

5) I could dump COMMAND.COM into a disassembler and find out where he
   sets the AL register before a call to DOS function 3D hex and alter
   it from binary 0 (compatibility mode) to 010 (Deny Write) mode

Obviously the first three are out, so what I'd like you to do, if you
don't mind, is to forward this note to the technical staff and/or 
programming team and suggest #4.

As for me, I will most likely try and patch COMMAND.COM myself and
see what happens!

Thanks for your time Julia.  I really appreciate it.  All along, I
thought that the problem was with the record locking function call,
it never dawned on me to look into how files are opened!  (Yeah,
I try and avoid DOS at all costs -- especially programming in it! :-)

If you want, I can let you know if my diving into the COMMAND.COM 
binary proves me correct and the problem disappears. I probably won't
get to this until next week.

======== cut here =======

So, I guess this posting is an  OBFhack (Obligatory future hack!)

-- 
Weave's 10/90 recipe for insanity: Take one programmer, divide into portions: 
10% programming consisting of equal parts in systems and applications plus
admin duties broken down into 20% for Unix, 5% PRIMOS, 10% hardware & network, 
35% DOS/Windows, 10% news, 5% postmaster, and 5% babysitting IRC kiddies.
Newsgroups: alt.hackers