Application of sparse L1 pursuits to resolution enhancement. - Page 1
Forum section: electronics / general

author: rhb
author_rank: Super Contributor
timestamp: December 22, 2018, 03:38:29 am
content: The application of sparse L1 pursuits to thermal camera images should provide an order of magnitude increase in resolution at nominal computational effort.In 2004 Emmanuel Candes and David Donoho wrote a series of papers which are the most important papers in applied mathematics since Norbert Wiener's work in the 1940's. Until the appearance of wavelets, all digital signal processing was based upon Wiener's work. Donoho and Candes developed wavelets on steroids.What they showed was that you could set up a matrix inverse problem, Ax=y where the number of unknowns was very much larger than the number of equations and that if you sought an L1, least summed absolute error, solution, *if* a sparse solution existed, it was the solution to the NP-hard L0 problem. That is, it was the best answer if you tried every possible answer. For problems of any size, that is impossible. The universe will cease to exist before you find an answer for even small problems.This is a very hot topic under the name "compressive sensing". It's also the basis of the solution that won the Netflix prize, blind source separation and a long list of other major developments in applied math over the last 15 years.Richard Baraniuk at Rice and his students took compressive sensing to its ultimate extreme in the form of a single pixel camera. What they did was use a single pixel photodetector to create an image by collecting multiple samples of the photodetector output with different spatial masks applied. A matrix of sub-pixels were blocked off, the photodetector output recorded as a y value, and then another. After enough samples had been collected, they solved Ax=y and produced an image.The attached paper summarizes that work. The original paper appeared in the IEEE Spectrum and IEEE is very aggressive about sending DCMA notices to anyone who posts it. So they wrote the attached paper.The mathematical justification of sparse L1 pursuits is the most difficult mathematics I have ever read. Fortunately, actually using the technique is easy. You set up the equations and solve them using any of several freely available packages. I've done all my work using GLPK, the Gnu Linear Programming Package.In the case of applying sparse L1 pursuits to increasing the resolution of a thermal camera, rather than using a single pixel and a mask, one can simply delete random pixels from a series of images.I've not tried this yet. I have seen the 512 x 512 Lena image reproduced very well from 20% of the pixels. I spent about 3 years reading the mathematical literature on sparse L1 pursuits. So I am reasonably confident I understand how and why it works.I already have more projects than I can ever finish. And this is not high on my "To Do" list. But I was asked to start a thread on the topic. I'm happy to offer guidance on how to implement it. Anyone asserting it can't be done will be directed to read the literature. It's much easier to do than it is to explain. One of Donoho's mathematical proofs ran for 15 pages! I am not a mathematician and cannot possibly explain mathematics as well as the original authors who write first class English prose.There has been an immense amount of work done on the subject, particularly at Stanford. I got caught up with the work to around 2014-2016, but have not followed it since then. After about 3000 pages I'd had enough. I suspect that a significant amount of work has been done on generating HD and 4K UHD images from lower resolution archival footage.

author: eKretz
author_rank: Frequent Contributor
timestamp: December 22, 2018, 09:19:40 am
content: Interesting, thanks for the information. Hadn't heard of this yet.

author: rhb
author_rank: Super Contributor
timestamp: December 22, 2018, 01:10:00 pm
content: Sidebar note:I stumbled across it in 2013. I was using Basis Pursuit following Mallat's discussion in the 3rd ed of "A Wavelet Tour of Signal Processing" to solve inverse problems in 1D heat flow and getting excellent results. One day I realized I'd been taught that you could not do this. I had long training in traditional DSP including 4 years pursuing a PhD under one of Wiener's students. So this *really* got my attention.I'd been put off by the wavelet hype from people who did not understand the mathematics and made exaggerated and inaccurate claims. I'd bought a number of books on the topic, but quickly concluded that for what I was doing traditional DSP methods following Wiener were the appropriate tool for most applications. I had Mallat's 2nd and bought the 3rd when it came out. I used it as a reference when topics from wavelet theory came up, but never read he book through. I'd read Mallat's explanation and go back to what I was originally reading.As it happened, "A Mathematical Introduction to Compressive Sensing" by Foucart and Rauhut appeared in 2013 so I ordered a copy. Wiener's classic work, "Extrapolation, Interpolation and Smoothing of Stationary Time Series" was known during WW II as "the yellow peril" for the difficulty of the mathematics and the yellow cover denoting a classified research report. It's trivial relative to Foucart and Rauhut's monograph. I was trained as a scientist, not as a mathematician. Foucart and Rauhut is a pure applied mathematics, a long series of proofs. So an order of magnitude more challenging than Mallat. After the first reading of F&R I read Mallat all the way through and then read F&R a 2nd time.I then started reading the original papers by Candes and Donoho. Both of them write beautifully and with some general idea of what it was all about I found them fairly easy reading. Their papers present an idea in a couple of pages of simple English prose. Then they proceed to present a rigorous mathematical proof. So someone without the background to read the mathematics can still understand the paper.After I wrote the initial post, I was playing with the HT-18 and noted that traditional DSP would do a very good job of cleaning up the images and removing row and column noise artifacts simply by doing a 2D FFT, applying a mask filter and an inverse transform. That's *very* easy to do if you have a UI that lets you pick polygons on a 2D presentation of the FFT and either keep or kill the interior of the polygon. It can all be automated, but there is a lot to be said for keeping a human in the loop making the key decisions.I spent 30 years as a research scientist/programmer. Mostly fixing other people's mistakes though I did write a fair bit of original code to solve particular tasks for which no tool was available. At 65 I'm much more interested in doing things with hardware than writing more software other than FOSS DSO FW. But I'd love to help build a FOSS package for processing thermal imagery following the latest developments in DSP.As I remarked earlier, you do *not* have to understand all the mathematics in order to use the tools. So if someone is interested in pursuing this, I'd be delighted to advise on the mathematical details.The biggest obstacle is language. Ultimately everything rests upon certain concepts from traditional DSP, so some familiarity with that and matrix algebra is essential. Building an image enhancement package for thermal cameras would mostly require writing bits of code to connect the major pieces. I'm fairly familiar with a lot of the relevant code. I actively supported the SU Seismic Unix package for about 15 years, so I know that code base intimately.

author: Bud
author_rank: Super Contributor
timestamp: December 22, 2018, 01:31:27 pm
content: Isn't it about sampling the original signal and not about post-processing of a already sampled signal ?

author: rhb
author_rank: Super Contributor
timestamp: December 22, 2018, 02:15:38 pm
content: So back to the actual subject.Sparse L1 pursuits solve Ax=y where y is the data, A is a matrix describing the problem and x is the solution vector. I have exclusively used the simplex method in GLPK for my work to date. It's very robust and it is quick enough for most problems. There are faster algorithms derived from the theory of regular polytopes in N dimensional space among other things, but I don't know of freely available code for those.There are two fundamental constraints:The columns of A must possess what is called the Restricted Isometry Property.The solution vector x must be sparse. Most of the elements of the solution vector must be zero.Explaining the RIP requires a basic grasp of classical DSP. If the cross correlation of two columns is near zero they are said to have low coherence. The RIP requires that any combination of columns must have a low coherence with any combination of the remaining columns. The RIP is essential to the mathematical proof, but computing it is NP-complete. So in practice it cannot be computed.So what you do is set up Ax=y and attempt to solve it. There will be one of two results. You will get an answer in L1 time or it will wander off in search of an L0 solution and you won't get an answer. You will almost always get a sparse solution. The easiest proof of this comes from the theory of regular polytopes in N dimensional space.In the jargon of mathematics this is called a convex optimization problem. We are seeking the point on the exterior of a cloud of points (in computational geometry this surface is called the convex hull) which is closest to an arbitrary plane. As it happens as the number of dimensions increase, the probability that a point lies on the convex hull increases.The number of dimensions is the length of the x vector.Mallat's description of basis pursuit is the easiest to understand. The columns of the A matrix form a dictionary of *all* the possible solutions you want to consider. This was how I stumbled across this. I computed a large matrix of solutions to the 1D heat equation for the purpose of solving inverse problems of fluid flow in porous media. So I was solving problems with 30-50,000 unknowns and 30-100 data points. GLPK would do one in a few minutes, so I could solve thousands of these overnight.Based on my prior experience solving inverse problems using L2, least squared error, I expected to get a smear of results around the correct value. Much to my astonishment I'd get the exact answer with maybe 1-2% of the total answer being incorrect values. For input data I was adding together a couple of columns of A with different x coefficient values. So the correct solution was N-2 zeros and two non-zero values. Typically I'd get 4-5 non-zero values, but they were very small relative to the large values.In the same manner as is commonly done solving eigenvalue problems using singular value decomposition, one can apply a threshold to eliminate the smaller values.In retrospect, it turns out that sparse L1 pursuits were being done for a long time in many fields with many different names. It's rather like a magician's bag. Every time you turn it inside out it's a different color. What has happened in the last 15 years is a solid mathematical basis for this has been rigorously developed. It's the intersection of a tremendous number of lines of analysis in mathematics.So long as you meet the two primary constraints you can solve any problem for which you can write the A matrix. There is a constraint on how many samples you need. Unfortunately it's an upper bound so it's difficult to get an actual value. But experience shows that for an arbitrary time series it's 10-20% of the number of samples dictated by the Nyquist criterion.This is being actively used to reduce MRI acquisition times and develop real time MRI video imaging.

author: eKretz
author_rank: Frequent Contributor
timestamp: December 22, 2018, 02:24:39 pm
content: It seems to me after reading the PDF attached to the OP that this could be used in a "superresolution" fashion similar to FLIR's UltraMax or similar methods from other manufacturers by taking multiple lower res images and extrapolating those using a custom solution that has yet to be developed. That may perhaps be similar to what those are based on in the first place.

author: rhb
author_rank: Super Contributor
timestamp: December 22, 2018, 02:30:19 pm
content: That is precisely the idea.

author: Bud
author_rank: Super Contributor
timestamp: December 22, 2018, 02:38:17 pm
content: Why such a complicated answer to my simple question. Superresolution is still using a combination of multiple _different_ images, they are not exactly the same. So is the described single pixel camera.

author: rhb
author_rank: Super Contributor
timestamp: December 22, 2018, 03:49:56 pm
content: I was just finishing writing that when you posted. There is no "save as draft" option. And your question was extraneous the purpose of that post. I'd very much like for this to lead to some software, so I'm trying to develop the concepts so that other people can participate indoingthis.Now to your question:Consider collecting multiple images of a stationary scene in the noise free case. All the images are identical. So if randomly masking multiple images will produce super resolution, then so will randomly masking the same image.I don't yet know how to write the A matrix, but that's a literature research project. So it should not be difficult.Even after 5 years, this whole thing still blows my mind. The whole idea that you can discard most of the data and get a better image boggles my mind.

author: Vipitis
author_rank: Frequent Contributor
timestamp: December 22, 2018, 11:02:07 pm
content: At first I was very sceptical. But given that they insist on this - it may be the solution to some problems. In the PDF they stated hyperspectral imaging for example.Right now I cannot understand most of it. I get the general idea but it's hard to believe I am just 19 and haven't finished school yet. But it's something very interesting that I might follow with my further education.But given the possible advancements for thermal imaging, what is possible and what isn't given the working code and maths Backend. Because Integration time is very low and you could just randomly selected a low number of pixels, if you tune your sensor correctly you may be able to achieve very high framerate at a full resolution with only ready some pixels. The model of a single pixel camera is like 40 years behind scanner cameras that use mirrors and one or two pixels only, I don't think we will go back from microbolometers to scanners.It also doesn't have the appearance of solving all the problems of letting our 60x80 or 320x240 sensors make nice 20MP images.How different is this to deconvolution or interferometry, you only collect your actual data from one point for the imaging idea?

author: rhb
author_rank: Super Contributor
timestamp: December 23, 2018, 01:02:00 am
content: Don't feel bad about not understanding all of it. This is an MSEE thesis topic or pretty close.If you go into EE it will be *very* important. But you've got so much math to learn to get there that it will take an MS to get there. However, if you already know about deconvolution, the Wiener prediction error filter, then you're *way* ahead of where I was when I was 29.In 1982 I had just started work as a geophysicist at Amoco after getting an MS in igneous petrology. I'd spent 9 months looking through a petrographic (polarized light) microscope at 54 thin slices of rock. They promised many weeks of training when I accepted the job, but then I had to process seismic data for 9 months and interpret it for 9 months before I got my first and only 2 week course in reflection seismology. I spent most of my evenings with a stack of books desperately trying to figure out how to do my job. But I enjoyed it so after a couple of years I decided to go back and try for a PhD in reflection seismology.The important thing about the single pixel cameras is it is *not* scanning. The concept is appearing in commercial products such as the TI DLPS based spectrometer although they call it Hadamard sampling.Going from 76,800 pixels to 20,000,000 is probably not possible. There probably aren't enough uncorrelated permutations of 76,800 pixels.There is a lot that can be done to improve the image from cheap thermal cameras like the HT-18 I bought just using classic Wiener DSP processing. I don't know anything at all about the thermal imaging world. Thermal imaging was treated for a few days in a remote sensing course I took 35-40 years ago as a tool for geological mapping.I haven't paid much attention to thermal cameras because I could not justify buying one. So I have no idea what level of image processing is being used. I looked at the sticky thread by JoeC, but as I don't know German I don't know how much it does. I got the impression it was mostly focused on calibration for accurate temperature measurements.Is there a package that does 2D FFT filtering of thermal images? If not, that's the place to start. There is a lot of seismic processing code which can be used as is, though the seismic jargon would make it difficult to understand by someone interested in thermal imaging. So building a thermal imaging system and just using the numerical routines would be more generally useful.Deconvolution will let you correct for image distortion caused by the lens and it will let you slightly increase the BW of an image, but you run into the problem of dividing by zero rather quickly. That's why you add some white noise to the data before computing the decon filter.Interferometry is a measurement technique rather than a signal processing method.The attached image was taken with my HT-18 with the lens cover closed. Is there software in the thermal imaging community that will remove the noise artifacts *without* averaging? If there isn't, then that's the obvious place to start. It's classic Fourier domain filtering. It's not trivial, but it's not hard either. Given such an image, one computes a filter which equalizes the amplitudes of all the rows and all the columns.I'm certain that there are all manner of complications to that rather simplistic statement, but it's pretty routine DSP.

author: Vipitis
author_rank: Frequent Contributor
timestamp: December 23, 2018, 10:23:11 am
content: In astrophotography we filter away noise using darkframes with the lens/telescope covered to just get the sensor readout noise at the same temperature and integration time then your light frames. But this is again averaged and subtracted. Cameras with a shutter do something like this automatically, but as far as I understand it it's again averaging and subtracting.

author: rhb
author_rank: Super Contributor
timestamp: December 23, 2018, 12:56:58 pm
content: Astrophotography has gotten *very* sophisticated, particularly the process of taking thousands of frames and aligning and stacking them. I need to find a recent professional level monograph on the subject. Although the mathematics is the same, every field has its own jargon. Also, a lot of times the popular understanding of what some software package is doing is a simplification of the actual process.As I remarked in the sidenote earlier. I want to play with hardware. So yesterday I was feeding a 5 nS pulse to a quartz crystal 10 million times a second and looking at the waveform. I also made a 1.5 V continuity tester. I blew up my Instek GDS-2072E and I want to fix it. I think I just blew the NAND flash chip. But I needed a continuity tester that was safe around 1.8 V logic. The pulse from the 33622A was a nice symmetric Gaussian. In seismic it would be called "zero phase". After applying it to the xtal it was asymmetric with a steep leading edge and a drawn out trailing edge. An absolutely perfect demonstration of a minimum phase response. I'd expected some ringing, but there was none at all.Everything in this thread will require GLPK, so I suggest downloading it and reading the manual. My preferred method has been to use GMPL which is a variant of AMPL. I write a program that creates a large GMPL data file and then call glpsol. This isolates the equations to a small file where they are easy to modify. With everything in ASCII text files, it makes debugging easier.The experiment I'm proposing is to solve for weighting coefficients for each row and column so that after multiplying each row and each column by the appropriate weight, the L1 error is minimized.That will not eliminate all the noise, but it will treat a part of the noise in a manner more in line with the physical source of the noise. The residual noise should be random 1/f, shot and other known noise processes. Once the noise is purely random, hopefully Gaussian, then one applies averaging of multiple images.In reflection seismic processing one goes through the data, transforms the data to a domain where the noise is coherent and can be filtered out, removes it and then proceeds to the next noise process. It is routine work to take data which has no discernible signal content and produce very good images.The best reference on classical Wiener DSP is "Random Data" by Bendat and Piersol. You may be able to get it in PDF form from your school. The most recent edition is the 4th which will be the last as Alan Piersol passed away. It's beautifully written and very comprehensive.

author: RoGeorge
author_rank: Super Contributor
timestamp: December 23, 2018, 02:20:30 pm
content: A little offtopic maybe, but I wanted to ask about it some time ago, when you mention compressing sensing in another topic: Is this compress sensing about exploiting the incredible "immunity to noise" of the sparse arrays?Please don't send me to read the 15 pages of demonstrations, math is not my true loveTo explain what I call "noise immunity": When I browsed the papers you mentioned there (was only diagonally reading, without even trying to follow the math), it seemed to me the key ingredient there is coming from the use of the sparse arrays. I heard of sparse arrays for the first time only accidentally, while learning about a new type of AI (Artificial Intelligence), based on HTM (https://en.wikipedia.org/wiki/Hierarchical_temporal_memory). I know from there one miraculous property of sparse arrays is these arrays are what I call "very resilient to noise (or errors)".To make an idea of the numerical performance, here is a video with some nice simulation for noise resilience of sparse arrays:https://youtu.be/Ub1fE-bAroA?t=442- to watch only the last 5 minutes or so, in order to understand the context for the numerical examples from the next videohttps://youtu.be/ZDgCdWTuIzc?t=0- numerical values plugged into simulator, and a first feel about how amazing Sparse Distributed Representations (SDR) can be.Is the "sparse L1" and "compressed sensing" about the same miraculous property of the sparse arrays?Did you noticed other spectacular properties while following the math from those papers you mentioned?

author: Bud
author_rank: Super Contributor
timestamp: December 23, 2018, 02:36:17 pm
content: Quote from: rhb on December 23, 2018, 01:02:00 amAn SVD (Singular Value Decomposition) based algorithm might do it, but unlikely in real time. I do not know of a practical software that may exist but the theory makes sense.

author: rhb
author_rank: Super Contributor
timestamp: December 23, 2018, 05:31:26 pm
content: HTM is something else again. It looks very much like the work Danny Hillis did over 30 years ago. Every 20-30 years people start reinventing the past. Which would be OK if they didn't then repeat all the same mistakes.In the equation Ax=y, the A matrix is dense. It is the x vector which is sparse. Immunity to noise depends upon how you write the constraints for the L1 minimization.I am not looking at this as a real time process. At least not until I've seen how long it takes. What I'm after is 660x480 or better resolution out of a 220 x 160 sensor. Tests of sparse L1 pursuits in other contexts suggest that 1100 x 800 resolution from a 220 x 160 sensor is probably realistic. 2200 x 1600 *might* be possible, but that is the upper limit of what I think possible. And I'm doubtful about that.But yesterday I wrote a piece about ultrasonic detection of cracked SMD parts saying I didn't think it could be done. But some simple experiments this morning proved me wrong. I still don't know if it can be done, but it now looks fairly likely it can. I just need a sub-nanosecond pulse source.Unfortunately, the HT-18 saves the image in a single layer. I'm going to ask them to consider changing that so that it is possible to extract the image without the annotation.

author: Vipitis
author_rank: Frequent Contributor
timestamp: December 23, 2018, 07:01:21 pm
content: getting a good starting place is important the best possible position would be a raw file that gives you 14bit(or 16 on the high end)pixels for every single one of the 35200 in your case or 4800 on my end and a tiny bit of meta data on the end of the file. FLIR offers the tools in different SDKs to give you something like this, but their first party software interpolates my 80*60 to a 320*240 That is horrible blurry for no reason.Given that most real world images have far less then this in data and you can collapse this into like 8bit without losing data it still gives you quite a bunch of data to compute that is not easily compareable to the two dimensional data that is shown as example of like 50 data points. Maybe a single row/column would work. A single row spanning 52° fov is reasonable sparse but not randomly samples at all. It is very even and every sub selection of a larger number of pixel is still in this even construct. Therefore I question the possibilities given a single stationary capture as input data. Maybe data over time is an option, but then you are at the current idea using superresolution like ultramax or a 3D reconstruction of a still scene. Yet again you only use the data available and don't generate new data in between.My interest in thermal cameras is mostly artistic and not at all for inspection or detection where datapoints are important to be exact and calibrated and noise is manageable. I strife for something that gives a noiseless image that looks analog and is very sharp. Resolution is the main tipping point and a device that is able to iterate a still scene over time is the dream as long as it's cheap.My academic pursues are more after the idea of interferometry and synthetizing a larger sensor and aperture by using multiple simultaneous captures with multiple sensor if different shapes and sizes. A current idea I am writing down right now even comebines thermal and visible images from multiple sensors to give out two images of high resolution, visible and thermal by finding correlations in fine detail and edges and using 'sparse' thermal pixels to apply gradients against it.If working with the HTi imagers is difficult, this forum will have someone to help you. We have managed to get raw data off all the dongle imagers that came up in the most recent 5 years using Linux(some on Pi) or Windows.I can supply radiometric jpgs from my 80*60 FLIR sensor, with the links on how to get low level raw data out of them.

author: Bud
author_rank: Super Contributor
timestamp: December 23, 2018, 07:10:45 pm
content: @rhbPeople reported snatching Flir E4 for $500 or something. If you have a vested interest in having a fairly good source of images just get an E4, it creates layered images which you can separate later or create just a thermal single layer images (thermal and visual separate) the E4 has a setting for it, and of you use my Liberation process you can make images with no measurements overlay or logo. Stop toying with lower grade gadgets that give headache and leap straight to your research.

author: rhb
author_rank: Super Contributor
timestamp: December 24, 2018, 12:54:05 am
content: I don't really have a need for anything better than the stock HT-18. I started this thread at the request of @ultrapurple who has not shown up.I spent 3 years pretty much full time learning about sparse L1 pursuits. I've done a bunch of stuff with it, but at this point I really don't have a lot of interest in pursuing image enhancement. I did that for 30 years and am throwing away the worn out T shirts from the annual professional convention. And I only used those for grub work clothes. So one of the T shirts added to the scrap cloth pile was from 1996.That said, it's a cool topic and if someone wants to work on it, I'm quite happy to advise and assist as needed. I worked out the L1 constraint equations for equalizing row and column errors this morning, but still need to test them.This is serious work, but it's not research. If I were on contract I'd be estimating $25-50K to implement this. If Flir is offering software resolution enhancement it is highly likely this is how they are doing it. Five years ago I would have jumped at a contract to do implement this, but with recent changes in my personal circumstances I'd pass. I'd rather play with this:https://www.eevblog.com/forum/projects/crack-detection/and similar small diversions.I've also spent close to $2k tooling up to write FOSS DSO FW for Zynq based scopes. So I am only interested in side projects that don't require large amounts of my time. Half a day to a day working out the solution to a problem for someone who wants to implement a FOSS image enhancement package for thermal cameras a couple of times a month, no problem. More work than that, sorry, but I'll pass. I won't learn anything new from doing it.In summary, if you want to do this and would like a PhD level advisor I'll do that. But that's as much work as I'm willing to put into it.

author: IanB
author_rank: Super Contributor
timestamp: December 24, 2018, 01:20:54 am
content: Quote from: rhb on December 22, 2018, 03:49:56 pmBecause you can't?You have to consider information theory and observability. You cannot create information out of nothing, or extract information that is not present in the source data. What you can do is interpolate, and create the illusion of data that was not there to begin with.This whole subject smacks of free energy and perpetual motion. Only it is wrapped up in so many words and so much mathematical jargon that the empty hole in the middle cannot be seen.There is real mathematics and solid theory, but it embodies a truth that is much more mundane and much less miraculous than some of the amazing words would make it appear.

author: rhb
author_rank: Super Contributor
timestamp: December 24, 2018, 12:21:49 pm
content: Ahh, grasshopper, you failed to heed my warning. As you insist it can't be done I must refer you to the mathematical literature. You can skip the math part unless you are interested in why. But read the introductions to these papers:https://statistics.stanford.edu/research/most-large-underdetermined-systems-linear-equations-minimal-l1-norm-solution-also-sparsesthttps://statistics.stanford.edu/research/compressed-sensinghttps://statistics.stanford.edu/research/extensions-compressed-sensingNB: Stanford has completely redone there website so that it is possible to search for reports by author and year. That interface is here:https://statistics.stanford.edu/resources/technical-reportsIf you still want to argue it can't be done, Candes and Donoho are in Rio to accept the 2018 Gauss prize, so you'll probably have to wait until mid-January to explain to them why it can't be done.Candes was at Caltech, so you'll have to go here:https://statweb.stanford.edu/~candes/publications.htmlto get his papers from 2004.It is important to note that Shannon applies to theinformationcontent. Three samples of a pure sine wave are all that is needed to convey the information.Please reread my first two posts to this thread.Edit: The statement quoted by IanB was with reference to the denoising formulation of compressive sensing.

author: rhb
author_rank: Super Contributor
timestamp: December 24, 2018, 12:57:17 pm
content: For the benefit of anyone interested in serious work in this topic these are what I consider the important monographs for this thread:Conduction of Heat in SolidsCarslaw and JaegerOxford Science Publications2nd ed 1959Random DataBendat & PiersolJohn Wiley and Sons4th ed 2010A Wavelet Tour of Signal ProcessingStephane MallatElsevier3rd ed 2009Linear ProgrammingRobert VanderbeiSpringer3rd ed 2008A Mathematical Introduction to Compressive SensingFoucart and RahutBirkhauser1st ed 2013For the benefit of the casual reader I'll post scans of anything relevant to the discussion for which original papers are not available.With the exception of Carslaw and Jaeger, I consider these monographs essential references for anyone in science and engineering. They constitute the basis of modern applied mathematical analysis. While B&P and Mallat treat Fourier theory, it's rather thinly. I use the 2nd ed of "The Fourier Transform and its Applications" by Bracewell, but it is out of print in the US though a later ed seems to be available in India. I have probably a dozen books on the Fourier transform, but Bracewell is almost always the one I use.The dictionary of graphs of transform pairs in the back of Bracewell lets you solve most problems with cartoons rather than calculus and algebra.

author: Vipitis
author_rank: Frequent Contributor
timestamp: December 24, 2018, 03:43:16 pm
content: To me a geometric proof is the best. You can find people who share this kind of feeling and invest thousands of hours into presenting it (Grant Sanderson). So if you know of any nice visuals that AIDS in the explanation and proof of this miracle, please send then first.

author: rhb
author_rank: Super Contributor
timestamp: December 24, 2018, 04:24:26 pm
content: Trust me, you don't want the geometrical proof. It is based on the theory of regular polytopes in N dimensional space. It's really cool, but it's not any easier.The canonical monographs are:Convex PolytopesBranko Grunbaum2nd edSpringer-Verlag 2003Lectures on PolytopesGunter ZieglerCorrected and update 7th printingSpringer 2007The initial papers relevant to sparse L1 pursuits are these:https://statistics.stanford.edu/research/neighborly-polytopes-and-sparse-solution-underdetermined-linear-equationshttps://statistics.stanford.edu/research/counting-faces-randomly-projected-hypercubes-and-orthants-applicationshttps://statistics.stanford.edu/research/counting-faces-randomly-projected-polytopes-when-projection-radically-lowers-dimensionhttps://statistics.stanford.edu/research/high-dimensional-centrally-symmetric-polytopes-neighborliness-proportional-dimensionJared Tanner is at Oxford now. I'm sure there is more work.FWIW I took a BA in English lit, an MS in geology and then went into geophysics. I lost my financial support after pursuing a PhD for 4 years, so I went back to work. Twelve years at university was enough. I'm still amazed that I managed to reach the point that I could read Foucart and Rauhut, Grunbaum, etc. Those are mathematical monographs intended for mathematicians. So just learning the lexicon is a lot of work.My motivation was very simple. Iknewyou couldn't solve problems that I was solving easily. So I had to find out why. Three years reading 3000 pages of mathematics without anyone to talk to is a bit frustrating. And no one who understands this has time for a 65 year old. They want someone who is 20 something.

author: Vipitis
author_rank: Frequent Contributor
timestamp: December 24, 2018, 11:36:20 pm
content: You linked like 200 pages and that's already an uncomprehendable task for someone without a higher math education.But I like the 3rd resource you link. Donoho 2004 it has lots of graphs and even some systematic images that give a good idea of what is possible. I prefer things I can see over things to read.

author: coppice
author_rank: Super Contributor
timestamp: December 25, 2018, 12:50:21 am
content: Quote from: IanB on December 24, 2018, 01:20:54 amYou seem to be confusing data with information. In most situations you can throw away lots of data and lose no information. That's what lossless compression is all about. In things like images you can carefully throw away a huge amount of data and lose little information. That's what lossy compression is all about.

author: IanB
author_rank: Super Contributor
timestamp: December 25, 2018, 01:02:02 am
content: Quote from: coppice on December 25, 2018, 12:50:21 amThis is not true. When you throw away lots of data you are also throwing away lots of information. Lossy compression works by selectively throwing away the least important information and keeping the most important information.You can understand that this is true by looking to the workflow used by professionals. In image processing, the professional will always go back to the raw image capture when producing new derived images. Similarly with music, the studio engineer will always go back to the original master recordings when producing new derived works.Lossy compression is a viable process because humans are not able to make use of all the received sensory information that comes in through the eyes and ears, so you can remove what the human physiology will filter out or miss and the subjective experience will still be good. The same is not necessarily true if a human being is not the intended consumer of the data.

author: coppice
author_rank: Super Contributor
timestamp: December 25, 2018, 01:11:18 am
content: Quote from: IanB on December 25, 2018, 01:02:02 amData is NOT information. If the data from a signal source looks random you can't reduce its size. Most data doesn't look random, though, and the information content of the data is quite sparse. You can compress most signals quite a lot, while losing no information at all. A lot of what makes the data from many signal sources look nearly random is actually just noise fuzzing up the data. Some mildy lossy compression schemes heavily suppress that noise, and actually clean up the signal, even though there may also be a little loss of information.

author: IanB
author_rank: Super Contributor
timestamp: December 25, 2018, 01:39:04 am
content: Quote from: coppice on December 25, 2018, 01:11:18 amFirst we were talking about lossy compression, and here you don't seem clear whether you are talking about lossy compression or lossless compression. Every other sentence you are hopping from one to the other. You need to be clear about what you are referring to. Lossless compression exploits redundancy in the source material. Lossy compression discards information deemed to be of low value to the consumer. They are quite different.Keep in mind that information is in the eye of the beholder. Whether data is information or not depends on how and why it will be consumed. That "noise" that you are suppressing from the signal might actually be the real information content inserted by steganography. What you think of as the clean signal may just be the decoy to mislead you.Never forget that all data is information. Even if the data is noise, it conveys information about the nature and character of that noise, and allows deductions about the noise sources present in the transmission path.

author: rhb
author_rank: Super Contributor
timestamp: December 25, 2018, 02:18:59 am
content: RTFM, Read The Fine Mathematics.Really, the introductions are easy reading. All the rest is just the fine print. For this discussion you can skip the fine print. If the fine print matters I'll point it out.Shot noise, 1/f noise, popcorn noise, etc arenotinformation unless you want to quantify them in order to characterize the performance of a device.The distinction between what is signal and what is noise depends entirely upon the task at hand. In this case we're trying to suppress visual noise artifacts and increase resolution.Candes was the one who got the ball rolling with the paper on exact reconstruction. That isnotlossy compression. It is lossless. And the applications of sparse L1 pursuits are so broad it would take several feet of graduate level books to just give an overview. So it is very hard to discuss sparse L1 pursuits without bouncing all over the place.I had to read 3000 pages to get to the point I could see a picture in my head. Sometimes that's what you have to do. Reality doesn't fit in a 15 minute YouTube video. The papers I linked are the easiest things to grasp. I strongly recommend that you read the introductory prose and look at the pictures first. That's what I would have done if I had not started out with Foucart and Rauhut which is long on equations and very short on pictures. It was a great relief when I started reading the original papers, except perhaps for the one with the 15 page proof. I think that made even Donoho's eyes glaze over.Most people fail by not trying. The thing that distinguishes PhDs and SOCOM operators is they don't quit. Failure just means you have to start over. You only stop trying when you are dead.

author: coppice
author_rank: Super Contributor
timestamp: December 25, 2018, 02:49:32 am
content: Quote from: IanB on December 25, 2018, 01:39:04 amWhy do you keep saying that, when the most basic principal of information theory is that it is bogus? Data is just a bunch of bits. Information is what you don't know, and can't figure out from other information you have. In the great majority of cases it is a subset, and often quite a small subset, of the data you are presented with.

author: rhb
author_rank: Super Contributor
timestamp: December 25, 2018, 12:54:48 pm
content: Quote from: coppice on December 25, 2018, 02:49:32 amA very important principle in programming is never to specify more parameters than information. Redundant information leads to bugs. Hopefully they crash the program, but all too often they lead to very subtle and difficult to diagnose errors in the results.The seminal paper was "Communication in the Presence of Noise". RFTM attached.NB: This thread is on the subject of mathematics applicable to thermal imaging. I started it by request following some comments I made in the HT-18 thread. From the little recent work I've read about thermal imaging it does not appear generally to be as sophisticated as astrophotography. In any case, sparse L1 pursuits are driving the state of the art in many disciplines. There are many opportunities to make major advances.

author: coppice
author_rank: Super Contributor
timestamp: December 25, 2018, 03:59:18 pm
content: Quote from: rhb on December 25, 2018, 12:54:48 pmThat's a good guiding rule, as long as its not followed cult like. Some values take a lot of compute to recreate each time they are needed, so calculating them once, and passing them around to all the things which need them, is a sensible efficiency move.

author: rhb
author_rank: Super Contributor
timestamp: December 25, 2018, 04:20:25 pm
content: Flir has the suppression of row and column noise blanketed with patents, however, I could not find any papers characterizing the sources of row and column noise.The most recent patent I found was granted in 2016.I've seen mention of hacks to increase the resolution of Flir cameras. Does anyone have any information about how Flir is handling resolution enhancements?

author: Vipitis
author_rank: Frequent Contributor
timestamp: December 25, 2018, 05:07:12 pm
content: Flir has 3 tricks.The main one is to use a higher resolution sensor and dumm in down in software to sell the same hardware as a cheaper model with a different model name. Meaning a whole line of Ex cameras are basically the same hardware and clever forum members found different methods of liberating the full potential by editing the firmware and configurion files. This enables only 1:1 sensor resolution, not beyond.Next up is UltraMax, which is basically just superresolution by shaky hands implemented in their high end handheld models of the Txxx series. Effectively increasing the image resolution 4x by combining 4 images and giving you actual, real data. Some visual cameras use their sensor stabilization to do a "sensor shift" and move the sensor by 0.5 pixel pitch around when you camera is on a tripod to effectively increasing your resolution using the same method.This is only practical on devices with a high framerate. ThermalCamera+ for FLIR One/CAT S60 has some form of superresolution built in as well.The third trick they do degrades their images by artificial interpolating into a higher resolution for no apparent reason. This causes blurring and fake data. This is very prominent in the low resolution Lepton devices like FLIR One, when you check an image in FLIR Tools or export a .cvs for example.For actual image enhancements based I maths, FLIR had different 'features' based on what kind of device they sell you.MSX is their patent to overlay an visible image with edge detection(in my mind wrongly implemented as off axis optics could fix the parallax issues and a synthetic image reinterpretation using image recognition and applying gradients of temperature with bases in the thermal pixels would make this an actually useful feature) it mostly covers up the horrendes low resolution on their consumer porducts.DDE is their digital detail enhancement patent technology found in devices for surveillance and security like military or hunting scopes/monoculars/vehicle based systems. It is a combination of standard histogram stretching and sharpening.they also we're caught adding fake noise to low end devices to make her top enf look better. This works similar too the resolution lock down and has been bypassed by forum members.As of any actual resolution enhancements with math tricks or computational imaging, nothing too my knowledge - but remember that this technology find military application first and the development might be much farther then publicly know. I could think of reconnaissance satellites using such advanced algorithms to get down to resolutions near Dawes limit.

author: IanB
author_rank: Super Contributor
timestamp: December 25, 2018, 07:04:05 pm
content: Quote from: Vipitis on December 25, 2018, 05:07:12 pmRight, but the achievable resolution is always going to be limited by the physics of the imaging system. Mathematics can help you achieve the maximum potential of the system, however to go beyond that would be over-unity, and nobody should rationally believe that to be possible.

author: Bud
author_rank: Super Contributor
timestamp: December 25, 2018, 09:33:49 pm
content: Quote from: Vipitis on December 25, 2018, 05:07:12 pmThey also induce substantial amount of artificial noise, which the liberation process gets rid of.

author: rhb
author_rank: Super Contributor
timestamp: December 25, 2018, 10:09:13 pm
content: I'm going to drop out of this thread. Between Flir's lawyers and DoJ's lawyers I might easily find myself with a lot of unwelcome problems.From the dates of the Flir patents on suppressing row and column noise it is clear that the signal processing aspect of thermal imaging is not very well developed in the private sector. I'm certain that Raytheon and other DoD suppliers are more advanced. But that is ITAR territory and I have no wish to go there in a public forum.I find it frustrating to do this, but there are times when it is best not to speak of certain things.

author: IanB
author_rank: Super Contributor
timestamp: December 26, 2018, 12:04:07 am
content: Quote from: rhb on December 25, 2018, 10:09:13 pmWith respect, you haven't gone anywhere in this thread. First we have vague, unspecific claims of miraculous mathematics. Now we have vague, unspecified fears of government persecution.Maybe you could explain, in plain English, as if to a high school student, what is the unique and special value of "sparse L1 pursuits"? That would garner much more useful discussion than pointing people to long and technical mathematical papers without saying what it's for or how it's helpful.Color me skeptical, but this thread has always been in danger of entering tinfoil hat territory, and you just took it there.

author: helius
author_rank: Super Contributor
timestamp: December 26, 2018, 12:33:08 am
content: IanB, with all respect, I think it is you who is derailing the thread. Why do you insist that something should be explainable in "plain English"? You remind me of the crackpots who dismiss string theory because it cannot be explained in "plain English". Or an infamous Usenet troll who refused to accept complex numbers unless they could be "explained in terms of apples."You also err in your claims about "workflow used by professionals." It is now possible to perform edits to individual instruments within a mix using Melodyne and no longer necessary to go back to a multitrack master recording. It is also common workflow to perform photo editing at a lower resolution than either the RAW file or the final output, with plugins like ON1 Resize. These are both applications of wavelet transforms. Wavelets are fundamental tools for modern engineering and it doesn't matter that you request plain English explanations.

author: eKretz
author_rank: Frequent Contributor
timestamp: December 27, 2018, 01:50:17 am
content: I think the misconception is in the fact that most people consider an individual pixel as only able to record or represent a single color and luminance level. It seems that is beginning to change. Here's another relevant link:http://news.mit.edu/2017/faster-single-pixel-camera-lensless-imaging-0330

author: IanB
author_rank: Super Contributor
timestamp: December 27, 2018, 02:20:21 am
content: Quote from: eKretz on December 27, 2018, 01:50:17 amCool. Very interesting.Also, that article does a remarkable job of explaining certain theory, and its application, and its value in simple, plain English without any complex mathematics. Kudos to the author of the article. The concept is straightforward and requires no magic. I predicted what the article was going to say before reading it.The slight half-truth in what you wrote () is that a single pixel is only able to record a single color and luminance levelat a single moment in time. If you allow time to be an additional dimension you can achieve more. But, this relates back to what I said in a previous post. To bring time into the equation more effectively and to achieve advances in imaging requires (very) clever physics and engineering. It is always the physics of the imaging system that defines and constrains what you can achieve.Mathematics, in and of itself, can only allow you to approach ever closer to the physical potential of the system (unity). Mathematics can never allow you to achieve more than that. Over-unity is not possible.

author: CatalinaWOW
author_rank: Super Contributor
timestamp: December 27, 2018, 02:58:52 am
content: While the optics is a fundamental element limiting resolution, there is some wiggle room in that limit. Most definitions of that limit boil down to the ability to use a simple amplitude threshold to separate two sources.To paint a simple word picture the optics response to a point source is more or less a gaussian shaped amplitude distribution. As two point sources are moved apart the amplitude response starts to show a dip and with enough separation that dip can easily be detected and separated from noise. This is the conventional definition of the resolution limit of an optic. Obviously the distribution from two point sources looks different from a single point distribution for any case other than when the two point sources are superposed. If the point distribution of the optic is known, you can determine the separation in two dimensions from the distribution of light from two sources, even if they are closer together than the "optical resolution".This gets more complicated when you sample the data with a detector and have a more general image than two point sources. I haven't digested the mathematics of this L1 stuff yet, but suspect that in essence doing this more general case is what is happening. With the benefit of computational efficiency. Not something for nothing, just wringing all the water possible out of the towel.

author: Ultrapurple
author_rank: Super Contributor
timestamp: December 29, 2018, 01:19:36 pm
content: Quote from: rhb on December 24, 2018, 12:54:05 amYes, sorry I haven't been around - I have been fully committed to family activities for the last several days.I am finding this discussion fascinating and can see how the sparse sampling techniques could make a major impact on thermal (or other intrinsically low resolution) imaging. One of my first thoughts was whether the regular array of pixels in a thermal camera could be treated as a special case of randomness and built upon from there. Similarly, using a 'randomly' coded aperture might be possible by selecting a random number of pixels from the array. (An interesting side-effect of this would be that dead pixels essentially cease to be a problem; whereas currently a sensor with 1% dead pixels is pretty much a reject, I can imagine sensors with50%dead pixels being perfectly usable. Manufacturing yield goes way up, prices come way down).The other exciting thing for me is that it appears that the basic principles have been shown to work, so it's 'just' a matter of turning the maths into applied software and cobbling together suitable hardware to run it on. So what if it takes a rack full of today's CPUs to run in real time? It won't be long before someone obeys Moore's Law and shrinks it into a single package. (At the start of what I laughably refer to as my 'career' I remember being really impressed with some real-time image processing avionics running on custom bit-slice processors encased in several boxes each the size and weight of two dozen bottles of wine. Nowadays the same image processing task would be considered trivial on a mid-range mobile phone - and, to my chagrin, I recently saw key parts of the system on eBay for peanuts). But I digress.The maths of all this is well beyond me but I am keeping up with the principles well enough to see at least some of the potential. I will continue to watch and learn with much interest.

author: Ultrapurple
author_rank: Super Contributor
timestamp: December 29, 2018, 04:27:59 pm
content: Quote from: CatalinaWOW on December 27, 2018, 02:58:52 amAll true. However, what happens if you add a time-of-flight dimension into the imaging system? Clearly, if the two point sources are (nearly) co-located at precisely the same distance from the sensor then ToF wont help. Furthermore, in thermal imaging we're normally considering emissions from the subject itself, rather than determining how it reflects an external energy source. I suspect a 'thermal flashgun with coded aperture lighting pattern' would be tricky to engineer but then again, some clever person will probably come up with pixels that are both phase- and frequency-sensitive, or otherwise unimaginably ingenious. I await developments with awe and excitement.Thermal imaging has come a long way in the last 20-30 years and I suspect it will develop even further.

author: rhb
author_rank: Super Contributor
timestamp: December 30, 2018, 04:05:08 pm
content: If you start with Candes and Donoho in 2004 and read the English prose introductions to their papers you'll have a very good grasp of what can be done. The proof details are important, but not particularly illuminating.All the rest is solve Ax=y using an L1 norm. The magic is in the properties of an L1 norm when the solution is sparse.The method is completely general so it applies in almost any endeavor. The constraints are that any combination of columns of A have low correlation with any other combination of columns. The other constraint is "sufficient" data. Realistically, you set up and attempt to solve the problem using a library call or the equivalent.I like using GMPL. The equations are in a small file which is hand edited. The data file is machine generated. The astrophotography crowd probably already has good algorithms for image enhancement. Some of which may be essentially a sparse L1 pursuit with a different name.