HDTenEightyP/NetNews huggingface dataset:
From: arms@cs.UAlberta.CA (Bill Armstrong)
Subject: Re: Wild values (was Reducing Training time ...)
Organization: University of Alberta, Edmonton, Canada
Date: Sat, 22 Aug 1992 19:56:17 GMT

kpfleger@NEURON.SIEMENS.COM (Karl Pfleger) writes:

>1 quick point and 1 wild idea:

>First, if one desires to avoid wild output values for certain regions of
>input space, one ought to have training pairs from that region of input
>space in the training set. The point about desiring certain behavior on
>0 to 1 and not including any training pairs from that region has already
>been made.

I agree that this would be desirable.  In some cases you don't have
the training points, and in other cases you could never get them
because there are just too many points in the space. High dimensional
real-valued data will always have the latter problem.

>Vaguely similar: since the inputs that will be thrown at the system in actual
>use will have some probability distribution based on whatever the system is
>doing, the training set should be generated by sampling the same
>distribution, or something as close to it as possible, NOT by picking a
>few values by hand or by using a lattice or points regularly spaced
>(unless that represents the distribution well).

The same example of non-safety will also work (neglecting numerical
errors) provided you take a random sampling of a few points according
to any distribution you like, except where you are not lucky enough to
have points in [0,1].  As I said, the fact that integer training and
test points are used is not significant.  The example of "wild"
behavior is not that easy to break down.


>I have a much more difficult time picturing wild values coming from a
>network trained on a significant number of random, real inputs than I do
>coming from a network trained on a handful of regularly spaced
>integers.

Try it and see.  You'll still have the problem.

>A wild idea for people trying to avoid wild values (e.g. for safety
>critical applications etc.): Once the network has been trained and the
>weights are fixed, ...

Your idea needs work, but calculating the aprtials and getting a
Lipschitz condition with a reasonable constant could work.

Thanks for your comments.

Bill

--
***************************************************
Prof. William W. Armstrong, Computing Science Dept.
University of Alberta; Edmonton, Alberta, Canada T6G 2H1
arms@cs.ualberta.ca Tel(403)492 2374 FAX 492 1071
From: doherty@vax.oxford.ac.uk
Newsgroups: comp.ai.neural-nets
Subject: Workshop,VLSI for NN & AI
Date: 12 Aug 92 10:02:58 GMT
Organization: Oxford University VAX 6620


                    3rd International Workshop on VLSI for Neural Networks
                              and Artificial Intelligence

                                  1-4 September 1992

                   at Jesus College, University of Oxford, England

For a leaflet and registration form, contact Maureen Doherty, CPD Centre,
Department for Continuing Education, University of Oxford, 1 Wellington Square,
Oxford. OX1 2JA  England.

Tel: +44 (0)865 270361. Fax:+44 (0)865 270284. Or leave your postal address by
e-mail, Doherty@uk.ac.oxford.vax.
Newsgroups: comp.ai.neural-nets
From: demers@cs.ucsd.edu (David DeMers)
Newsgroups: comp.ai.neural-nets
Subject: Re: Network Inversion
Date: 23 Aug 92 04:27:03 GMT
Organization: =CSE Dept., U.C. San Diego

Thanks for the list of references!

In article <al.714491544@jargon> al@jargon.gmd.de (Alexander Linden) writes:
>demers@cs.ucsd.edu (David DeMers) writes:

>>In article <BtCqHo.AtH.1@cs.cmu.edu> tjochem+@CS.CMU.EDU (Todd Jochem) writes:
>>>I'm looking for references to network inversion papers. The basic idea

>The method Todd asked for is gradient descent in input
>space with respect to some error measure at the output.

That's not apparent from his post.

If a network is computing y = f(x), then to my mind inversion
is finding f^{-1}(y).  This is normally one-to-many, and
if x is in X a compact n-dimensional manifold and y is in Y,
a m-dimensional manifold, the inverse will be a finite
set of n-m dimensional manifolds (assuming f is smooth). 

If you have an inverse problem, that is, find x such
that f(x) = y, you can solve it differentially, which seems to be 
what you are calling "inversion".  This can be done in several ways,
either pseudo-inverse methods or using the adjoint operator,
which is what you get by backpropagating delta_y to the inputs.

This is pretty well known in the control literature.

The differential methods result in a solution on the same
manifold you started, since they can't drive through a 
critical point (where the Jacobian of the forward map loses rank).

...

>I did not see that this method calculates the gradient in input space.

I fail to see why a gradient is mandatory...
Aren't we seeking a solution x to f(x) = y?

>A direct calculation of the input seems not possible because it is 
>often a many to one--mapping. Also the thresholds are not considered in
>his work...

Sure, it's possible - you just have to select one.  It's a
regularization problem, really.  


-- 
Dave DeMers			 ddemers@UCSD   demers@cs.ucsd.edu
Computer Science & Engineering	C-014		demers%cs@ucsd.bitnet
UC San Diego					...!ucsd!cs!demers
La Jolla, CA 92093-0114	(619) 534-0688, or -8187, FAX: (619) 534-7029
Newsgroups: comp.ai.neural-nets
From: arms@cs.UAlberta.CA (Bill Armstrong)
Subject: Re: Reducing Training time vs Generalisation
Organization: University of Alberta, Edmonton, Canada
Date: Sat, 22 Aug 1992 20:12:07 GMT

henrik@mpci.llnl.gov (Henrik Klagges) writes:

>sef@sef-pmax.slisp.cs.cmu.edu writes:

>>Well, since you keep pounding on this, I will point out that in most
>>backprop-style nets after training, almost all of the hidden units are
>>saturated almost all of the time.  So you can replace them with sharp

>Same in our experiments. The decision trees being built do benefit a lot 
>from the remaining nonlinearities, though (smoother decision surfaces-
>really 8-).

Isn't it a question of smooth vs discontinuous?  If you don't need
continuity, then you don't need the sigmoids at all.

>>Myself, I prefer to think in terms of parallel hardware, so lazy evaluation
>>isn't an issue.  Yes, sigmoid unit hardware is a bit more expensive to
>>implement than simple gates, but I don't need nearly as many of them.

>It is not terribly expensive - a 256 entry table is usually enough. Pipe
>lined access to such a lookup table can be made at one lookup/cycle at a 
>pipe stall of less than 5 (if not much better, hihi). Moreover, weight
>accumulation/update are matrix operations, while lookup is only a vector
>operation. It is no bottleneck at all.

OK, but looking up things in RAM is just using combinational logic.
Your ALN competitor will be finished the entire NN computation in
the time it takes to do two or three of these lookups.

Another problem is you seem to assume that numerical accuracy is not
any issue at all.

--
***************************************************
Prof. William W. Armstrong, Computing Science Dept.
University of Alberta; Edmonton, Alberta, Canada T6G 2H1
arms@cs.ualberta.ca Tel(403)492 2374 FAX 492 1071
Newsgroups: comp.ai.neural-nets
From: hokenek@watson.ibm.com (E.Hokenek;085396)
Subject: Re: Using discrete weights to solve networks with I/O pattern pairs
Date: Wed, 12 Aug 1992 18:28:51 GMT
Disclaimer: This posting represents the poster's views, not necessarily those of IBM
Organization: IBM T.J. Watson Research Center
Keywords: neural network weight discretization, discrete weights

In article <1992Aug11.182631.1511@uni2a.unige.ch>, osherson@uni2a.unige.ch writes:
|> > From: eskfoo@ntuvax.ntu.ac.sg
|> > Organization: Nanyang Technological University - Singapore
|> 
|> > I have recently found a way to implement forward networks with binary 
|> > I/O patterns using discrete weights [ -1, 0 , 1]. I have tried out this 
|> > using the XOR problem and the weights are as follows:
|> >
|> > [data omitted]
|> >
|> > The above result is obtained using Kaunaugh map analysis. Other examples
|> > with 4 input neurodes and 1 output neurode have been tested and found to
|> > be working.
|> >
|> > Just wondering if this method is applicable for all cases. Please comment
|> > or provide suitable references to the above mentioned.
|> 



You might be interested in the following reference!
Erdem


Reference:
Erdem Hokenek
Analysis and synthesis of Neural Networks using Linear Separation,
33rd Midwest symposium on Circuits and Systems, Calgary, Canada,
Volume 1, pp. 25-28, Aug.1990.
From: heading@breeze.rsre.mod.uk (Anthony Heading)
Newsgroups: comp.ai.neural-nets
Subject: Re: Metrics in Self Organizing Maps
Date: 13 Aug 92 16:26:01 GMT
Organization: Defence Research Agency

Whoops. In an earlier reply, I wrote that for the city-block distance,
delta w = k sgn(x-w)x. This is indeed wrong, as a couple of my
colleagues have been kind enough to point out. The correct rule is
delta w = k sgn(x-w) and the x I included is utterly spurious. Back
to primary school arithmetic lessons for me, I think...

Raman Rajasekhar notes that this isn't really a terribly good
update rule, since determining k is tricky for continuous updates.
I agree. It seems to me that there is a fairly obvious batch
update method though, which we obtain by requiring
sum over patterns (sgn(x-w)) = 0. This criterion is satisfied
when w is the centre of the data in the sense that the number
of samples which are respectively larger and smaller are equal.
(For odd numbers of data items w should sit squarely on the
central one.) Each dimension can be treated independently here.
I can't see any easy way to find the correct k for a continuous
scheme at the moment. Any ideas?

Anthony

-- 
Anthony Heading  (heading@hermes.mod.uk)
DRA Malvern, UK
Any opinions expressed are my own and in no way
represent those of my employer or any other organization.
Newsgroups: comp.ai.neural-nets,can.ai,comp.ai,comp.compression,comp.theory
From: lima@moon.ral.rpi.edu (Pedro Lima)
Subject: Question:Measuring complexity of CPU-time?
Organization: Rensselaer Polytechnic Institute, Troy NY
Date: Thu, 20 Aug 1992 19:44:02 GMT

The definition of Kolmogorov Complexity of a string s as the smallest program p
that when presented as input to an Universal Turing Machine U outputs s 
may cause practical problems since U is allowed to run p for any length of
time as long as it eventually terminates. This time may of course be too
long to be practical (think of a NP-complete problem). Thus, some authors
define a time-bounded complexity where essentially U is restricted to
terminate within T steps leaving s as output.

My question is: is there any problem to use in the definition of K-compl.
the number of time-consuming operations (say flops) instead of the length
in bits of the machine code of a program and thus consider computational
time as a measure of complexity itself?

Thanks

Pedro Lima
lima@ral.rpi.edu
From: wltam@cucs17.cs.cuhk.hk
Newsgroups: comp.ai.neural-nets
Subject: Pros. and Cons. of NN. for general application.
Date: 19 Aug 92 03:20:07 GMT
Organization: Dept of Computer Sci, CUHK

Hello all,

	This posting want to collect some NN. practitioners' idea
about the pros and cons. of using NN. for certain application .

	For example, some requires a lot of training time,..some 
is quite domain/application-specific,...some does not gurantee to
provide answers in some circumstances.....etc.

Best Rdg.,
Vincent Tam. 8-)
Newsgroups: comp.ai.neural-nets
From: yq01+@andrew.cmu.edu (Yulin Qin)
Newsgroups: comp.ai.neural-nets
Subject: Looking for a post-doc or RA position in neuroscience
Date: 17 Aug 92 11:35:23 GMT
Organization: Doctoral student, Psychology, Carnegie Mellon, Pittsburgh, PA


           Looking for a post-doc or RA position in neuroscience
 
	I am a graduate student in the department of psychology at Carnegie
Mellon University, with Herbert A. Simon as my advisor, and will get a
Ph.D. soon.
 
	The research I have done focuses on higher level cognition, which leads
to my desire to understand how cognition may actually be physically
implemented in the human brain. I am very eager to engage in
neuroscience research and gain experience in this field. Thus, salary
and other benefits are of secondary concern to me. What I primarily want
is a chance to learn and work in the area of  neuroscience after I
graduate from CMU. If I cannot be a post-doctoral fellow, I would like
to be a research assistant or something like that. 
 
	My publications, with my advisor and professor Tom Mitchell (computer
science, CMU), are related to the following areas:
	1) Scientific discovery (Cognitive Science (1990). 14, 281-308.);
	2) Human and machine learning (Proceedings of the Fifth Florida
Artificial Intelligence Research Symposium (1992). 235-239);
	3) Mental imagery and problem solving (Proceedings of the Twelfth
Annual Conference of the Cognitive Science Society (1990). 646-653; 
Working Notes of AAAI Spring Symposium on Reasoning with Diagrammatic
Representations (1992). 18-23; Proceedings of the Fourteenth Annual
Conference of the Cognitive Science Society (1992). 1050 - 1055).
 
	I have a Master's degree in computer science (Artificial Intelligence
and Statistical Pattern Recognition), and have a strong training and
five years experience in applied statistics (e.g., experiment design and
analysis).
 
	Detailed resume and sample publications will be be sent to anyone who
are interested in my application. Suggestions for where I may be able to
find a position in neuroscience are very much appreciated.
 
Yulin Qin  (Mr.)
Department of Psychology
Carnegie Mellon University
Pittsburgh, PA 15213
 
Phone (412)268-6284
E-mail: qin@psy.cmu.edu
Newsgroups: comp.ai.neural-nets
From: arms@cs.UAlberta.CA (Bill Armstrong)
Subject: Re: Characterizing learnable functions
Organization: University of Alberta, Edmonton, Canada
Date: Wed, 12 Aug 1992 17:27:00 GMT

tmb@arolla.idiap.ch (Thomas M. Breuel) writes:

>In article <1992Aug12.112845.1060@cs.tu-berlin.de> async@opal.cs.tu-berlin.de (Stefan M. Rueger) writes:

>   >Maybe consumers of NNs should ask for and insist on Lipschitz conditions:
>   >   |output(a) - output(b)| <=  const * | a - b |
>   >
>   >for all a, b in the domain of the function. 

>   These Lipschitz-functions **are** continuous and thus covered  by the
>   results of Cybernko, described in my original article.

>Unfortunately, imposing a Lipschitz condition is not enough. While a
>neural network might be able to approximate any Lipschitz function,
>the number of training examples (the "sample complexity") you need for
>actually finding a "good approximation" is provably too large in many
>cases.

Excellent point.  If the training or test data is very sparse, you
would have to have an infinitesimally small Lipschitz constant to
specify what you want on the whole space or to determine that the
result of training is within spec.

On the other hand, if you look at the structure of a net and you try
to compute a Lipschitz bound at a given point, you would probably have
to use absolute values of weights, and make pessimistic estimates of
the derivatives of the sigmoids, so you would end up with a uselessly
large bound.

I wouldn't give up on the idea of Lipschitz bounds yet, but it looks
bleak.  The only alternative I can see is to assume piecewise
monotonicity of the desired function in the specification, and to
force piecewise monotonicity of the result of training.

AN IMPORTANT QUESTION: Can anyone think of any other straightforward
ways to get bounds on values *between* test points?  The application
of NNs in safety-critical areas is at stake.
--
***************************************************
Prof. William W. Armstrong, Computing Science Dept.
University of Alberta; Edmonton, Alberta, Canada T6G 2H1
arms@cs.ualberta.ca Tel(403)492 2374 FAX 492 1071
Newsgroups: comp.ai.neural-nets
From: fenner@engcon.marshall.ltv.com (JWFENNER)
Newsgroups: comp.ai.neural-nets
Subject: Re: Dynamically adding nodes, layers ?
Date: 12 Aug 92 19:29:07 GMT
Reply-To: fenner@engcon.UUCP (JWFENNER)
Organization: LTV MEG, Dallas, TX

In article <1992Jul31.132715.2858@cs.hw.ac.uk> writes:
>I hope someone can help me here,  I seem to recall reading somewhere
>about a learning algorithm which first checks to see if the new pattern
>can be learnt without overwriting old info, and if not, then adding new
>nodes as required.  I *DEFINITELY* read about this somewhere, unfortunately
>I dismissed this as being of little import to what I am trying to do.
>However, the world being as it is, this irrelevance has become relevant.
>So, if anyone can help me I would be most appreciative.

There are two networks that come to mind.  The first one is known as the
Hyperspherical Attractor Network (HAN), developed by Robert Kennedy of
the Naval Coastal Systems Center Research and Technology Department.  This
network adds & deletes nodes as necessary to learn.  It is a good pattern
classifier network. Reference:

The Hyperspherical Attractor Network: Concepts, Theory, and Application
Robert A. Kennedy
December 1990
Naval Coastal Systems Center Research and Technology Department
Panama City, FL

The second one I know of is called GMDH (Group Method of Data Handling). I
don't know much about it, other than the fact that it can add layers when
training.  I do know the book I saw it in was called NEUROCOMPUTING by 
Robert Hecht-Nielsen. 

Hope this is helpful.

-- 
==================================================================
= Jon Fenner (engcon!fenner@uunet.uu.net)                        =
= LTV Aerospace and Defense - Neural Network / AI Research Group =
==================================================================
Newsgroups: comp.ai.neural-nets,comp.ai,comp.compression,comp.theory
From: dzik@access.digex.com (Joseph Dzikiewicz)
Subject: Re: measuring complexity (was Re: Kolmogorov Complexity)
Organization: Express Access Online Communications, Greenbelt, MD USA
Date: Wed, 12 Aug 1992 15:08:57 GMT

In article <1992Aug10.115352.1@acad3.alaska.edu> fxsdb@acad3.alaska.edu (Scott D. Berry) writes:
>an
>assembly language program, for example, may have an infinite number of states.

Actually, unless you have an assembly language that allows addressing
infinite memory, even an assembly language program has a finite
number of states (a large number, admittedly, but still finite).

This lack of infinite memory is why digital computers are of equal 
computing power to a finite state machine and considerably less powerful
than a Turing machine.  Admittedly, a computer with megabytes of memory
is a finite state machine with a lot of states, but it's still not
Turing-equivalent.

Joe Dzikiewicz


Newsgroups: comp.ai.neural-nets
From: hoekstra@cs.utwente.nl (Aarnoud Hoekstra)
Subject: Articles wanted
Organization: University of Twente, Dept. of Computer Science
Date: Tue, 18 Aug 1992 07:58:15 GMT


Hello,

I am looking for an article which was presented at the IJCNN-91 in Seattle. Is there
anyone who has a copy of the article? Please let me know.

The reference :

Self-organizing neural network for spatio-temporal patterns. M. Hagiwara. IJCNN-91-Seattle
(Cat. no. 91CH3049-4). 8-14 july 1991. p 521-524 vol2.

Thanks in advance


Aarnoud Hoekstra
Dept. of Computer Science
University of Twente
hoekstra@cs.utwente.nl

:-) just keep on smiling :->
From: key@blacksea.csc.com (Gerry Key)
Subject: Question: Neural Nets and Data Verification
Keywords: Neural Nets NN Data Verification Environmental
Organization: Naval Ocean Systems Center, San Diego, CA
Distribution: usa
Date: Wed, 12 Aug 1992 16:48:18 GMT

I am a biologist and not a regular reader of this newsgroup.  Please  
respond directly via email.  I will post a summary to comp.ai.neural-nets  
if the responses warrant.  I poked around in the FAQs before posting this  
inquiry, but didn't find anything that clearly addressed my question. 

I recently saw (but did not read and now cannot find) an article on using  
neural network technology to verify large data
familiar with this technology and can you give me some
background information, products, etc.?

The application I have in mind is environmental survey data - everything  
from
databases, or the data matrices derived from them, tend to be large and  
and complex, and checking them is tedious work

I'm not so much concerned with numerical outliers that should be  
identified and evaluated during routine statistical analysis.  Rather, I'm  
looking for a
"quality control", if you will.  I'd also like a tool that can be used by  
the researcher (rather than requiring a programmer) and one that can be  
taught to recognize similar patterns in different data.  I'm not  
interested in conducting neural network research, so am looking for a  
commercial, off-the-shelf (COTS) product with at least nominal vendor  
support.  

A simple-minded example of the application I have in mind would be if the
variable measured during a survey was salinity, then the units associated  
with that measurement should be parts-per-thousand,
not pH, degrees Celsius, or number per square meter.  Similarly, the  
method used to count fish might be a visual transect, beach seine, or a  
fish trap, but not a pH meter, gas

I recognize that to some extent these tests can be performed by integrity
checks available through the load module of an RDBMS.  However, the
"rules" why some measurements need closer examination can be quite complex  
and subtle and not easily expressed as if-then-else logic.  

The data will come from a variety of sources, some instrumented and some  
not, but all will ultimately wind up in an SQL database.  Thus, a tool  
that could be run against such a database would be ideal. 

Any suggestions/pointers would be greatly appreciated.  
 
--Gerry Key
Computer Sciences Corporation
Internet:               key@csc.com
FAX:                    619.225.2502
Voice:                  619.225.2504
 
Newsgroups: comp.ai.neural-nets,comp.ai,comp.compression,comp.theory
From: choy@skorpio.usask.ca (I am a terminator.)
Subject: Re: measuring complexity (was Re: Kolmogorov Complexity)
Organization: University of Saskatchewan, Saskatoon, Canada
Date: Wed, 12 Aug 1992 17:35:46 GMT

In article <arms.713549466@spedden>, arms@cs.UAlberta.CA (Bill Armstrong) writes:
|> mather@sees.bangor.ac.uk (Paul Mather) writes:
|> 
|>  ... lots of stuff I agree with ...
|> 
|> >(*) PS (for the comp.theory people): Does an algorithm have to be finite?
|> >Consider an infinite set of random instructions spat out by some other algorithm
|> >running continuously on a Turing machine, say.  Would the infinite stream of
|> >executable instructions be classed as an algorithm?
|> 
|> The trouble is that all you need to define any function is a
|> (complete) table of input-output pairs.  If you allow this table as
|> (part of) the "instruction set", all of computability theory becomes
|> uninteresting.  So yes, normally algorithms have to be finite.

Why would computability theory become uninteresting?

|> That having been said, my gut feel is that mathematicians must have
|> looked at various transfinite numbers of "instructions" to get
|> functions on even larger spaces.
|> --
|> ***************************************************
|> Prof. William W. Armstrong, Computing Science Dept.
|> University of Alberta; Edmonton, Alberta, Canada T6G 2H1
|> arms@cs.ualberta.ca Tel(403)492 2374 FAX 492 1071

Henry Choy
choy@cs.usask.ca
Newsgroups: comp.ai.neural-nets,comp.ai,comp.compression,comp.theory