HDTenEightyP/NetNews huggingface dataset:
From: paulking@next.com (Paul King)
Newsgroups: comp.ai.neural-nets
Subject: neural nets and generalization (was Why not trees?)
Date: 24 Jul 92 02:45:18 GMT

arms@cs.UAlberta.CA (Bill Armstrong) writes:
> The question is: What is the advantage of making connections in a
> non-tree fashion?

sef@sef1.slisp.cs.cmu.edu (Scott) writes:
> 2. Why try to minimize the number of independent free connections in
>    the net?
>
> Training is indeed faster with more connections, but generalization
> is better with fewer connections.  The extreme case is seen in various
> image processing tasks (e.g. char recognition) in which it is almost
> mandatory to use shared connections or some sort of bottleneck -- else
> you would need many more training examples than pixels in the image.

Here is an additional observation on the issue of generalization.

The collection of training patterns for a neural net could be viewed as:

    possible input patterns  --> [Black Box] --> correct output patterns

Neural networks seem to operate on the assumption that what goes on
in the black box is rational and deterministic -- that a causal chain
of events transform an input pattern into an output pattern.  The
"goal" of the neural net is not only to memorize the input-to-output
mappings, but also to reverse engineer the causal chain of intermediate
states that presumably exists inside the black box.

Hidden units are interesting because they occasionally converge
on an intermediate representation that highlights regularities
in the behavior of the black box.  These intermediate states can
then be used as inferred information about internal states of the
black box, which can be useful as inputs to further networks.

As an example, consider handwriting recognition.  Mapping input
drawings to target characters might result in hidden-units that
correlate with subfeatures in the input drawings.  This ability
to map input drawings to subfeatures might then benefit a higher-level
system that maps subfeatures to words, bypassing the character
representation altogether.  One now has a network that can recognize
words even if certain letters are incompletely written.

Paul King
paul_king@next.com
From: jim@netlink.cts.com (Jim Bowery)
Newsgroups: comp.ai.neural-nets
Subject: Re: Quality of training data
Date: 22 Jul 92 14:14:38 GMT
Organization: NetLink Online Communications, San Diego CA

The selection of training data and the order in which it is presented
(assuming one is not using a batch mode) has been studied in education
theory by Zigfried Engleman and applied in classroom learning programs
quite successfully.  The theory is called "conceptual learning" and it
involves queueing the stimulus response pairs so as to time their
reintroduction to match the time when the student is most likely to 
begin to "forget" that particular stimulous-response pair.  I think
Marty Siegel at the University of Illinois (at least he used to be
 there) might have some references, although a person who might be
more conversant in the mathematics of the theory as applied to
stochastic approximations like neural nets would be Lou Dibello.

--                    
INTERNET:  jim@netlink.cts.com (Jim Bowery)
UUCP:   ...!ryptyde!netlink!jim
NetLink Online Communications * Public Access in San Diego, CA (619) 453-1115
From: sef@sef-pmax.slisp.cs.cmu.edu
Newsgroups: comp.ai.neural-nets
Subject: Re: re:need for unique test sets
Date: 27 Jul 92 13:56:53 GMT
Article-I.D.: cs.1992Jul27.135653.270071
Organization: School of Computer Science, Carnegie Mellon


    From: tom@cs.su.oz.au (Thomas James Jones)
    
    	I guess this is pretty difficult to answer, but what
    	basis do we have for expecting that the net should
    	(or can) learn a function we have 'in mind', when there
    	are infinite trivially different ways of mapping any
    	given data set?
    	If we have no organizational basis for our learning, then
    	what inherent organizational concepts do we expect the
    	net to extract?
    
Yes, it is pretty difficult to answer.  Any given architecture or algorithm
will be "biased" in favor of certain representations for a set of data, and
this bias may be better or worse for generalization to new problems drawn
from the same distribution.  The state of theory in this area can generally
be found in the COLT conference proceedings.  That's "Conference on
Learning Theory", and is not restricted to neural net models.

-- Scott
===========================================================================
Scott E. Fahlman
School of Computer Science
Carnegie Mellon University
5000 Forbes Avenue
Pittsburgh, PA 15213

Internet: sef+@cs.cmu.edu
Newsgroups: comp.ai.neural-nets
From: davisd@milton.u.washington.edu (Daniel Davis)
Subject: Re: need for unique test sets
Organization: University of Washington, Seattle
Date: Tue, 21 Jul 1992 22:40:19 GMT

I hope I can clear up this debate with a little specificity.

A couple guys say that all we need is independant sampling, while
someone else seems to think that one should not include the training
data in the test set.

Independant sampling is in fact all you need, but given the proper
context, it is also proper to say that one should not include any of
the training data in the test set.

Suppose you take 10000 independant samples. You use 5000 as your
training set. You would *not* select your test set from all 10000
samples, but instead, only from the 5000 not included in the training
set. If you selected test data from a random sampling of all 10000
samples, your test data and your training data would no longer be
independant. Of the data you have, only the 5000 previously unselected
data correspond to data independant of your training set. In this
sense, then, one should not include any of the training data in the
test data. 

However, it is *not* a problem if it happens that some of the 5000
previously unselected data are in fact repeats of the original
training data, as it is assumed that the original 10000 were
independant samples.

Buy Buy -- Dan Davis
Univ. of Washington, Dept. of EE, davisd@u.washington.edu
Newsgroups: comp.ai.neural-nets
From: elsberry@cse.uta.edu (Wesley R Elsberry)
Subject: IJCNN 92 Beijing Call For Papers *Extension*
Organization: Computer Science & Engineering, Univ. of Texas at Arlington
Date: Tue, 21 Jul 1992 20:11:16 GMT


		       IJCNN 92 BEIJING, CHINA
			  November 3-6, 1992

			   CALL FOR PAPERS

			  --- Extension ---

Papers for the upcoming IJCNN 92 Beijing conference will continue to
be accepted through July 31, 1992.  Papers should be submitted in
standard IJCNN format (maximum of six pages, etc.) to:

	Dr. Harold Szu
	9402 Wildoak Drive
	Bethesda, MD 20814

Dr. Szu indicates that because of the limited technical communication
between the rest of the international ANN research community and the
People's Republic of China, papers submitted may be original work *or*
review articles.

Researchers are encouraged to contribute on any topic related to 
artificial neural networks.

---
Wesley R. Elsberry
Sysop, CNS BBS 509-627-6267; Moderator Int'l NEURAL_NET Echo;
Newsgroups: comp.ai.neural-nets
From: arms@cs.UAlberta.CA (Bill Armstrong)
Subject: Re: Correctness of NNs
Organization: University of Alberta, Edmonton, Canada
Date: Mon, 27 Jul 1992 19:47:34 GMT

robert@cs.strath.ac.uk (Robert B Lambert) writes:


>     The individual neurons
>in the brain are causal devices. Their state is based on the history of firing
>of the cell and the history of received pulses.

I would like to agree with you.  If this is true, then you could use
an ALN model to try to learn to approximate this functionality
(boolean signals, discrete time), independently of any non-boolean
parts in the actual system, which merely serve to transform a history
of input pulses to the output pulse stream.  One of my questions is
just: does anyone see a problem in using ALNs this way, even if they
are not an accurate model of the neuron?

>>I will leave it up to the BP people to worry about the safety of their
>>systems.  Up to now, it seems they won't even admit there is a
>>problem.
>>

>I agree with you to a point. Simple NNs by there very nature are unreliable.

That's not my argument.  If we are talking about digital as opposed to
analog NNs, then they can be as reliable as any other computing
device, whether they are ALNs or the usual MLPs.  The problem I see is
that we train them on limited data, and testing will not show up all
the deviations from what we want, even if we can specify what we want.
It's the people who count on miracles that are the real danger in NN
technology.

>What is the future of ANNs? If they are to be used in any situation where a 
>fully definable input-output set exists, they have no future as this is the 
>application where conventional computer technology excels.

ALNS produce conventional computing technology! Namely combinational
switching circuits, which you could implement using today's cheap,
off-the shelf programmable logic devices. We could use ALN technology
to implement fully definable functions in a faster way.  I see that as
promising for numerical computation of the usual kind.  Here, what
applies to NNs does not apply to BP type nets, because the latter are
not fast enough to compete with conventional methods, while ALNs are.

 If ANNs are to be
>used for real world control and recognition tasks we must face up to the fact
>that such networks while able to give the best performance can never be 100%
>reliable as it is simply not possible to account for all possible inputs.

You have to have a spec, and you have to have a way for proving your
ANN is within spec.  So even in cases where you have astronomical
numbers of inputs, you can still prove an ANN is within spec.  The
major problem is when you have no spec, as in many recognition tasks.

Thanks for your thoughtful comments.

Bill


--
***************************************************
Prof. William W. Armstrong, Computing Science Dept.
University of Alberta; Edmonton, Alberta, Canada T6G 2H1
arms@cs.ualberta.ca Tel(403)492 2374 FAX 492 1071
From: atan@park.bu.edu (Ah-Hwee Tan)
Newsgroups: comp.ai.neural-nets
Subject: Re: Dynamically adding nodes, layers ?
Date: 31 Jul 92 19:31:59 GMT
	<ATAN.92Jul31122341@park.bu.edu> <zd3x2##@rpi.edu>
Organization: Boston University Center for Adaptive Systems
In-reply-to: floydb@sun01.admin.rpi.edu's message of 31 Jul 92 17:02:32 GMT


Belows are some of the references on ARTs and ARTMAPs:

\item Carpenter, G.A. and Grossberg, S. (1987a) A massively parallel
architecture for a self-organizing neural pattern recognition machine.
{\em Computer vision, Graphics, and Image Processing}, {\bf 37}, 54-115.

\item Carpenter, G.A. and Grossberg, S. (1987b) ART2:
Self-organization of stable category recognition codes for analog
input patterns. {\em Applied Optics}, {\bf 26}, 4919-4930.

\item Carpenter, G.A., Grossberg, S., and Reynolds, J.H. (1991) ARTMAP:
Supervised Real Time Learning and Classification by a self-organizing
neural network. {\em Neural Networks}, {\bf 4}.

\item Carpenter, G.A., Grossberg, S., Markuzon, N, Reynolds, J.H., and
Rosen, D.B. (1991) FUZZY ARTMAP: A Neural Network Architecture for
Incremental Supervised Learning of Analog Multidimensional Maps. {\em
IEEE Transactions in Neural Networks}, In Press. (A short version of
this appear in IJCNN-92 Baltimore III-309 to 314.) 

\item Carpenter, G.A., Grossberg S., and Rosen, D.B. (1991) FUZZY ART:
Fast stable learning and categorization of analog patterns by an
adaptive resonance system. {\em Neural Networks}, {\bf 4}.


As for the Cascade Correlation Algo, you could get the ps file
fahlman.cascor-tr.ps.Z by ftp archive.cis.ohio-state.edu.
It is in dir pub/neuroprose.

Enjoy,

AhHwee Tan
Dept of Cognitive and Neural Systems, Boston University
From: hcbarth@afterlife.ncsc.mil (Bart Bartholomew)
Newsgroups: comp.ai.neural-nets
Subject: Re: need for unique test sets
Date: 29 Jul 92 05:27:54 GMT
Organization: The Great Beyond

In article <arms.712338868@spedden> arms@cs.UAlberta.CA (Bill Armstrong) writes:

[stuff deleted]
>One more question please, that maybe you are in a position to answer
>as a result of your (.mil) domain.  I have done contracts for the
>Defence Research Establishment Valcartier, which showed that ALN ideas
>could be applied to FLIR images (ca. 1980).  I heard from someone
>attached to a foreign military research establishment last year that
>my stuff was actually quite popular in their country.  That surprised
>me a lot, since it didn't seem like the ALN idea had gotten very far.
>Do you know of any other non-classified work on ALNs that has been
>done for the military, that you could point me to?
>
>--
>***************************************************
>Prof. William W. Armstrong, Computing Science Dept.
>University of Alberta; Edmonton, Alberta, Canada T6G 2H1

	Um, no.  The only work I know of is from one of the
IJCNN (?'90/Washington, DC?) using NNs to classify (oops,
bad word :)) inputs from towed sonar sensors.
	Friend Armstrong, I meant no attack on you or even
ALNs.  I merely predicted your propensity to champion your
cause.  I suspect the rest of the group has already noticed
it. :)  I have no complaint; if you have something of value
that you believe in, who could do otherwise?  Note that I 
am experimenting with V2.6 to learn what I can from it.
	BTW, how the %$#@ do you get it to save the trained
net?  Nothing (Including RTFM, & reading the code) has worked
so far.  I get a parsing error from LFEDIT no matter what
I try.
	Bart
-- 
"It's not the thing you fling, the fling's the thing." - Chris Stevens
 If there's one thing I just can't stand, it's intolerance.
*No One* is responsible for my views, I'm a committee. Please do not
infer that which I do not imply.  hcbarth@afterlife.ncsc.mil
Newsgroups: comp.ai.neural-nets
From: tsioutsias-dimitris@CS.YALE.EDU (Dimitris Tsioutsias)
Subject: ALN vs BP
Organization: Yale University Computer Science Dept., New Haven, CT 06520-2158
Date: Fri, 24 Jul 1992 05:36:23 GMT

It seems that after the backprop fans, we have now the ALN ones. Why
each group (or any other that shows strong support) is trying to pass
its nets as the dominant ones? It's like we should stop looking at
anything else and concentrate all our efforts to the ALN's! Relax.

The field of NN is still new and noone should draw any strong conclusions.
We are still looking at different directions, trying various things,
testing trends, dynamics, determining parameters, in a few words
EXPERIMENTING. To claim that we have found the tool to work on
and reject the rest, could be premature and wrong at this point.

->dimitris
tsioutsias@cs.yale.edu

Newsgroups: comp.ai.neural-nets
From: bdbryan@eng.clemson.edu (Ben Bryant)
Subject: Re: comp.ai.pattern-recognition (was: Speech re
Reply-To: bdbryan@eng.clemson.edu
Organization: College of Engineering, Clemson Univ.
Date: Thu, 23 Jul 1992 01:28:05 GMT

In article 21975@Csli.Stanford.EDU, kawai@Csli.Stanford.EDU (goh kawai - n6uok) writes:
>Brendt Wohlberg suggests:
>
>   > why is there no separate group for general pattern recognition - 
>   > comp.ai.pattern-recognition say?
>
>Stephen L. DeKorte suggests:
>
>   > comp.ai.pattern-recognition sounds like an excelent idea.
>   > Anyone else interested in starting such a news group?
>
>I am a linguist (a phonetician, to be precise) working on speech
>recognition.  I would be very interested in starting a news group, either
>for pattern recognition in general or for speech processing in particular.
>My colleagues and I recently established a speech processing newsgroup in
>Japan -- fj.comp.speech -- and would like to see something similar here on
>Usenet.  
>
>----------------- speech technology and research, sri, menlo park, california
>--- Goh Kawai --- work:(415)859-2231   fax:(415)859-5984   home:(415)323-7214
>----------------- internet: kawai@speech.sri.com      radio: n6uok and jk1zyp

I am currently finishing up a Master's in EE with an emphasis in DSP and
PR as applied to speech recognition, and I would welcome such a group.

-Benjamin Bryant
<bdbryan@eng.clemson.edu>


Newsgroups: comp.ai.neural-nets
From: esmythe@fergie.dnet.ge.com (Erich J Smythe)
Newsgroups: comp.ai.neural-nets
Subject: Re: Thanks Neural Nets, and Brains?
Date: 24 Jul 92 16:33:17 GMT
Organization: GE Advanced Technology Labs

In article <arms.711907358@spedden>, arms@cs.UAlberta.CA (Bill
Armstrong) writes:
[...]
>First off, isn't it rather strange that the most widespread artificial
>model of neural operation. the multilayer perceptron, uses continuous
>quantities on its connections, while the dendrites and axons of
>neurons use "zero or one" type action potentials?  Until physiological
>psychologists start studying adaptive logic networks, can anyone
>expect much progress on understanding the brain?

Gee, I don't recall ever reading that dendrites and axons use "zero or
one" action potentials.  Most of what I've studied led me to believe
that "dendritic computation" was characterized by continuous
quantities of of membrane potentials.  A good deal of the neural
modelling I've read attempts to model and explain the rich behavior of
dendritic membranes.

Perhaps I've missed something.  Could someone direct me to the proper
references where the "zero or one" action potential is claimed for
dendrites?  I am, of couse, aware of spike potentials and the like,
but have never viewed the neuron as a logic gate where the sole unit
of information is the spike potential.  I always viewed information
transfer in the nevous system as being slightly more complex.

>
[...]
>***************************************************
>Prof. William W. Armstrong, Computing Science Dept.
>University of Alberta; Edmonton, Alberta, Canada T6G 2H1
>arms@cs.ualberta.ca Tel(403)492 2374 FAX 492 1071

-erich
Erich Smythe                            esmythe@fergie.atl.ge.com
GE Advanced Technology Labs
Moorestown, NJ
I don't speak for GE.
Newsgroups: comp.ai.neural-nets
From: uds@wsu-eng.eng.wayne.edu (Seetamraju Udaybhaskar)
Subject: Re: Learning what COULD be learned
Reply-To: uds@wsu-eng.eng.wayne.edu (Seetamraju Udaybhaskar)
Organization: Wayne State University, Detroit
Date: Wed, 22 Jul 1992 00:51:47 GMT

In article <arms.711759321@spedden> arms@cs.UAlberta.CA (Bill Armstrong) writes:
>
>If a tree isn't learning the required task, as shown by a lack of
>further improvement, then you can double the size of the tree for that
>particular output and try again.  Does this make it clear how the
>technique of independent trees would be used?
>
>--
>***************************************************
>Prof. William W. Armstrong, Computing Science Dept.
>University of Alberta; Edmonton, Alberta, Canada T6G 2H1
>arms@cs.ualberta.ca Tel(403)492 2374 FAX 492 1071


could you clarify, about how, with the number of inputs constant, the
size of the tree could be varied ?


Seetamraju Udaya Bhaskar Sarma
(email : seetam @ ece7 . eng . wayne . edu)
From: bill@nsma.arizona.edu (Bill Skaggs)
Newsgroups: comp.ai.neural-nets
Subject: Re: Neural Nets and Brains
Date: 23 Jul 92 20:56:14 GMT
	<arms.711907358@spedden>
Organization: ARL Division of Neural Systems, Memory and Aging, University of
	Arizona

arms@cs.UAlberta.CA (Bill Armstrong) writes:

   >First off, isn't it rather strange that the most widespread
   >artificial model of neural operation. the multilayer perceptron,
   >uses continuous quantities on its connections, while the dendrites
   >and axons of neurons use "zero or one" type action potentials?  

It is true that multilayer perceptrons use continuous-valued signals,
but there are many artificial models around that use binary-valued
signals, e.g. Hopfield nets, Boltzmann machines, Kanerva's Sparse
Distributed Memory, etc., etc..  As a matter of fact, the
McCulloch-Pitts model used binary neurons.

   >Until physiological psychologists start studying adaptive logic
   >networks, can anyone expect much progress on understanding the
   >brain?

Well, the McCulloch-Pitts model was used extensively, and very
productively, as a model of the brain, so this isn't too convincing.

Any abstract model includes some features and leaves out others.
Which features will turn out to be crucial can only be known in
retrospect.  There is no reason to think that continuous models are
incapable of shedding any light on nervous systems.  For theoretical
work they have some real advantages -- among the most important being
that they make possible certain learning rules, such as backprop, that
cannot be used with binary models.

"Let a hundred flowers bloom."

	-- Bill
From: Daniel.K.R.McCormack@cm.cf.ac.uk (Daniel K R McCormack)
Newsgroups: comp.ai.neural-nets
Subject: Refs wanted on DeltaBarDelta
Date: 27 Jul 92 15:50:25 GMT
Organization: University of Wales College of Cardiff, Cardiff, Wales, UK.

Hi,

	I've heard about a minimisation technique called 
DeltaBarDelta which is used for BP but I cannot find any 
references to it in the literature. Can anyone send me some
refs on this please ?

Many Thanks

	Dan
Newsgroups: comp.ai.neural-nets
From: dvoelker@leland.Stanford.EDU (David Voelker)
Subject: Re: Neural Networks and Computer Music
Summary: Reference for music and connectionism
Keywords: Neural Networks Computer Music AI Artificial Intelligence NN
Organization: DSG, Stanford University, CA 94305, USA
Date: Wed, 22 Jul 92 06:23:18 GMT

>In article <1992Jul14.000924.18805@gmuvax2.gmu.edu> phesse@gmuvax2.gmu.edu (Peter Hesse) writes:
>>
>>A resounding good-day and hello to all of you out there in
>>the world of internet and bitnet.  My name is Pete Hesse,
>>My first question, or "favor" to ask is if anyone has
>>suggestions on good "readable" texts dealing with neural
>>networks.  Also, does anyone know of publications, source
>>code, journal references, or FTP archives that discuss or have
>>information on previous projects dealing with the applications
>>of neural networks in computer music?

Here's one:
             Music and Connectionism
             Peter M. Todd and D. Gareth Loy, eds.
             MIT Press (1991)
-- 
=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=
|   Dave Voelker      Stanford University     dvoelker@leland.stanford.edu  |
=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=
Newsgroups: comp.ai.neural-nets